Inquiry Regarding mAP Accuracy Calculation for YOLOv8n on Raspberry Pi 5 & Hailo-8L

General
,
1

Hello Hailo Support Team,

I am currently working on a project that requires calculating the mAP accuracy for a YOLOv8n model.

My development environment is as follows:

  • Target Device: Raspberry Pi 5 (running Raspberry Pi OS, which is Debian-based)

  • AI Accelerator: Hailo-8L M.2 Module

  • Host PC: Ubuntu 22.04 (with the Hailo AI Software Suite and Hailo Model Zoo installed)

I am facing a challenge with the accuracy evaluation due to my host-target development setup. My Host PC is used for compiling the model (generating the .hef file), while the Raspberry Pi is the target for actual inference and evaluation.

I have attempted the following methods to measure the accuracy, but unfortunately, none have been successful:

  1. Running hailomz eval on the Host PC: This failed with a “Hailo device not found” error, as the host machine does not have the Hailo hardware.

  2. Running hailomz eval on the Raspberry Pi: This was not possible because the hailomz toolchain is not supported on Raspberry Pi OS (Debian ARM).

  3. Creating a Manual Evaluation Script: I tried to write a custom Python script for the Raspberry Pi using tools like pycocotools. However, implementing the complex post-processing logic and the full mAP calculation from scratch has been very difficult.

Could you please recommend the standard workflow or provide guidance on how to properly calculate the mAP accuracy for my YOLOv8n model in this host-target environment?

Thank you for your time and assistance.

2

Hi @minjoo_kim

Welcome to the Hailo community. At DeGirum (a SW partner of Hailo), we developed PySDK to simplify development with Hailo devices. One of the tools we developed is a model evaluator class that can help you with your evaluation. A detailed guide for evaluating Hailo models is at: Hailo guide: Evaluating model accuracy after compilation - Guides / Hailo Guides. Please see if you find this useful.

3

Hi Shashi,

Thank you so much for your reply!

The PySDK and model evaluator tool from Degirum look very useful for evaluating models within the Degirum Model Zoo.

However, my main goal is to evaluate the accuracy of the HEF files that I compiled myself from Ultralytics’ YOLOv8n models. This includes the following five types:

  • YOLOv8n-cls (Classification)

  • YOLOv8n-det (Object Detection)

  • YOLOv8n-seg (Instance Segmentation)

  • YOLOv8n-pose (Pose Estimation)

  • YOLOv8n-obb (Oriented Bounding Box)

It seems the Degirum evaluation tool can only evaluate models that are already in the Degirum Model Zoo. Could you please let me know if there’s an alternative script or guide available for evaluating the accuracy of a locally compiled HEF file?

Thank you, minjoo_kim

4

Hi @minjoo_kim

PySDK and the model evaluator tool are not limited to models within DeGirum model zoo. Any model compiled for Hailo can be integrated to pysdk and used. See Hailo guide 3: Simplifying object detection on a Hailo device using DeGirum PySDK - Guides / Hailo Guides - DeGirum Community how to integrate your own YOLO models to PySDK. Once you have the model, the model evaluator can be used.

5

Hi Shashi,

Thanks so much for the helpful advice! I really appreciate it.

I’ll give that a try and let you know how it goes.

Have a great day!

Best regards,

Minjoo

1 Like