Hailo-8 Benchmark score issue

Hello Hailo team,

Please review my questions related to Hailo-8.
We’re working to find reason why Hailo benchmark score was different from real score on our hardwares.

Our HW configuration

  • System 1 with Intel i3 CPU with Hailo-8
  • System 2 with Intel i7 CPU with Hailo-8

We tested Hailo’s reference hef model from model zoo.
we use same model with hailo benchmark reference.

  • resnet-50 v1
  • vit base
  • YOLOv5m
  • YOLOv8m
  • YOLOv11m

we got similar scores with resnet-50 v1 model…within 1% gap.
but YOLO model were lower over 20% gap.

our system works PCIe Gen3 x4 Lane well…
we used several drivers and other Ubuntu OS version…but same results.
we used same batch size…8.

Below is result.
Could you guide me for idea ??

yolov8m reference score is 139 but our score was 108.63. 22% lower
yolov11m 102 71.85. 29% lower
vit_base 107 97.44. 9% lower


I hope to get your advice ASAP…

thanks

Hey @bcmtechfy,

Thank you for catching this discrepancy. You’re right - the numbers indicate version 2.13, but they should be higher for v2.14. I’ll investigate this issue and fix it.

I’ll update you once this is resolved.

Best regards,
Omria

Hi Omria,

thanks for your fast update but I’m sorry that i can’t understand exactly.
What is the v2.13 and v2.14 ?
the benchmark scre in website is correct score? or wrong?
we used several driver version but same result…
we want to know how we can get same resuts.

thanks

Hi @jy.han,

Welcome to the Hailo Community !

For reference:

  • Version 2.13 (released October 2024) uses HailoRT 4.19
  • Version 2.14 (released January 2025) uses HailoRT 4.20

I’ll provide an update regarding the models soon, as we are currently reviewing them.

Great news - we’ve identified and fixed the FPS issue. The problem was with the HEF files for YOLOv8 and YOLOv11. We’ve now updated them to the correct versions that achieve the expected FPS.

Thanks for bringing this to our attention!