Hi @jungmin_jeon
Your three measurements agree with each other, but I think the pair you are comparing is not the experiment you want it to be. I pulled the zoo’s own compiled binaries and ran hailortcli parse-hef on them (this needs no device, it just reads the HEF), and the two models differ in more than where the NMS runs.
1. The two HEFs do not contain the same thing
======== hailo8l / yolov5xs_wo_spp
Network group name: yolov5xs_wo_spp, Multi Context - Number of contexts: 2
Output yolov5xs_wo_spp/yolov5_nms_postprocess FLOAT32, HAILO NMS BY CLASS(number of classes: 80, maximum bounding boxes per class: 80, maximum frame size: 128320)
Operation:
Op YOLOV5
Name: YOLOv5-Post-Process
Score threshold: 0.200
IoU threshold: 0.60
Classes: 80
Max bboxes per class: 80
======== hailo8l / yolov5xs_wo_spp_nms_core
Network group name: yolov5xs_wo_spp_nms_core, Multi Context - Number of contexts: 3
Output yolov5xs_wo_spp_nms_core/nms1 FLOAT32, HAILO NMS BY CLASS(number of classes: 80, maximum bounding boxes per class: 800, maximum frame size: 1280320)
yolov5xs_wo_spp prints an Operation block: that is a HailoRT net-flow op descriptor, i.e. the NMS is not in the network, it is handed to the host and HailoRT runs it on your CPU. yolov5xs_wo_spp_nms_core has no such block, it has an nms1 layer inside the graph. So hailortcli benchmark --hw-only on the first one measures a network with no NMS in it, and on the second one it measures a network with the NMS in it. The gap being “already there at the pure NPU-throughput level” is not a finding about NMS placement, it is what the two binaries are: the host variant’s NMS simply is not in that number.
2. And they are not even solving the same NMS problem
|
yolov5xs_wo_spp (host) |
yolov5xs_wo_spp_nms_core (on-core) |
| score threshold |
0.20 |
0.01 |
| IoU threshold |
0.60 |
0.45 |
| max boxes per class (from the HEF) |
80 |
800 |
| output frame |
128,320 B |
1,280,320 B |
| pretrained ONNX |
.../2023-04-25/yolov5s512.onnx |
.../2022-05-10/yolov5s512.onnx |
| fp32 mAP in the zoo |
33.18 |
32.72 |
A 20x lower score threshold and 10x the proposal budget is a much bigger NMS, and the output frame is 10x larger. On top of that they are two different checkpoints from two different dates, with different fp32 mAP, and their .alls differ in more than the engine (the host one carries allocator_param(width_splitter_defuse=disabled)). Whatever ratio you get out of that pair, it is not “cost of moving the NMS on-core”.
Small side note on the 800: the config json in the zoo says max_proposals_per_class: 100, but the shipped binary reserves 8x that. The only 8 in that file is input_division_factor: 8. I have not confirmed the two are related, and I would be glad if someone from Hailo could say what governs the reserved box count for engine=nn_core, because it drives both the NMS work and the output frame size.
3. The zoo’s other on-core model pays 1.79x, not 3.6x
v2.19.0 ships exactly two nms_core networks. The second one, yolov6n_0.2.1_nms_core, is configured sanely (80 classes x 100 boxes, score th 0.03) and is single-context on both architectures. Published batch=1 FPS:
| arch |
host NMS |
on-core NMS |
ratio |
extra per-frame period |
| Hailo-8L |
yolov5xs_wo_spp 206 |
nms_core 57.2 |
3.60x |
+12.63 ms |
| Hailo-8L |
yolov6n 356 |
nms_core 199 |
1.79x |
+2.22 ms |
| Hailo-8 |
yolov5xs_wo_spp 1139 |
nms_core 100 |
11.39x |
+9.12 ms |
| Hailo-8 |
yolov6n 1250 |
nms_core 237 |
5.27x |
+3.42 ms |
So the answer to your first question is: no, 3.6x is not a documented property of on-core NMS, it is a property of that particular build. Same generation, same 80 classes, a different on-core model costs half as much.
4. The cost behaves like a fixed per-frame slot, not like a fraction of the backbone
Read the last column across architectures. For a given build the extra period barely moves between Hailo-8 and Hailo-8L (12.63 vs 9.12 ms for yolov5xs, 2.22 vs 3.42 ms for yolov6n), while the network time itself changes by 3-4x. Between the two builds it changes by 4-6x, which is the direction their proposal budgets differ (800 vs 100).
That is the real mechanism behind what you observed, and it is not “NMS-core is bad for lightweight backbones”. It is that on-core NMS adds a roughly constant per-frame cost that does not shrink when the backbone does and does not shrink when the chip gets faster. As a ratio it therefore looks worst exactly where the network is cheapest, which is small backbones and the faster part. Your yolov5xs case is the worst corner of that grid, and it is made worse still by the 800-box config.
One extra 8L-specific effect: the on-core build needs 3 contexts where the host build needs 2, and both are single-context on Hailo-8. That extra context boundary is a plausible explanation for why the 8L delta (12.63 ms) is larger than the 8 delta (9.12 ms) for the same model, though I have not isolated it.
5. On your second question: the yolov8s pair does not test this
yolov8s and yolov8s_bbox_decoding_only both run their postprocess on the host:
yolov8s.alls: nms_postprocess(..., meta_arch=yolov8, engine=cpu)
yolov8s_bbox_decoding_only.alls: nms_postprocess(..., meta_arch=yolov8, engine=cpu, bbox_decoding_only=True)
Neither is engine=nn_core, so identical FPS (110/208 on 8L) is the expected result and says nothing about on-core NMS. There is no yolov8 on-core entry anywhere in the zoo to generalise from. In my own testing on DFC 3.33, engine=nn_core gets rejected at allocation for DFL-based heads, which is consistent with the zoo only shipping on-core builds for anchor-based yolov5 and for yolov6n.
6. When I would actually use on-core NMS
Your framing in question 3 is the right one. On-core NMS does not buy throughput, it buys host CPU and a small fixed-size output. It is worth it when the host CPU is the scarce resource: several streams or several models on one Pi, a CPU already busy with capture, scaling and encoding, or a host where you want the postprocess off the critical path and deterministic. On a Pi 5 running a single model with three cores idle, host NMS wins on FPS, because HailoRT’s op runs on another thread and hides behind inference, which is exactly what your app shows when its end-to-end number lands on top of --hw-only.
If you do want on-core, the knobs that matter are nms_scores_th, max_proposals_per_class and the class count, not the backbone. The zoo carries yolov5xs_wo_spp_nms_config_1class/_5class/_10class/_20class variants precisely because that is where the cost lives.
The experiment I would run
Recompile yolov5xs_wo_spp_nms_core with the same NMS config as the host build (nms_scores_th: 0.2, nms_iou_th: 0.6, max_proposals_per_class: 80) from the same 2023-04-25 checkpoint, and re-measure both. That is the controlled A/B, and my expectation is that most of your 2.3x goes away.
Two caveats on everything above. I could not re-measure this on silicon, my Hailo-8L is physically out of the machine at the moment, so sections 1-2 and 5 are read off the shipped binaries and configs and section 3-4 uses the zoo’s own published numbers, not my own. And the zoo measures on an i5-9400 with PCIe Gen3 x4, while you are on Gen2 x1: your 44.84 FPS against the zoo’s 206 for the same model on the same chip is a 4.6x shortfall that is worth understanding on its own, and it is also why you see 2.3x where the zoo table says 3.6x.
Everything here is reproducible without a device:
curl -sSf -O https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.19.0/hailo8l/yolov5xs_wo_spp.hef
curl -sSf -O https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ModelZoo/Compiled/v2.19.0/hailo8l/yolov5xs_wo_spp_nms_core.hef
hailortcli parse-hef yolov5xs_wo_spp.hef
hailortcli parse-hef yolov5xs_wo_spp_nms_core.hef