Help compiling custom YOLOv8 ONNX to HEF (NMSConfigPostprocessException, DFC 3.33.0)

General Discussion
, , ,
1

Hi,

I’m trying to compile a custom YOLOv8 model (single class = “rat”) to HEF for Hailo-8L using the Hailo Dataflow Compiler 3.33.0 on an AWS EC2 instance.

Environment:

  • Hailo DFC: 3.33.0

  • Python: 3.12 (venv)

  • OS: Ubuntu on AWS EC2 (t2.large)

  • HW arch: hailo8l

  • Model: custom YOLOv8, exported to ONNX (rat_yolo_best.onnx)

What I did

1. Parse ONNX → HAR

hailo parser onnx rat_yolo_best.onnx --hw-arch hailo8l

This produced:

rat_yolo_best.har

hailo har info shows:

  • NMS Meta Architecture: Yolov8

  • Files in HAR:

    • rat_yolo_best.hn
    • rat_yolo_best.alls
    • rat_yolo_best.nms.json
    • rat_yolo_best.npz
    • rat_yolo_best.original_model_meta.json
    • rat_yolo_best.HARFileNames.POSTPROCESS.onnx
    • rat_yolo_best.metadata.json

2. Extract NMS + model script

hailo har extract rat_yolo_best.har --model-script-path rat_yolo_best.alls
hailo har extract rat_yolo_best.har --nms-config-path rat_yolo_best.nms.json

3. Optimize

hailo optimize rat_yolo_best.har \
-–hw-arch hailo8l
-–use-random-calib-set
-–model-script rat_yolo_best.alls
-–output-har-path rat_yolo_best_opt.har

It starts modifying YOLO layers, then crashes:

NMSConfigPostprocessException:
The layer rat_yolo_best/conv43 doesn’t have one output layer

Earlier, I had also seen:

AllocatorScriptParserException:
Cannot infer bbox conv layers automatically.
Please specify the bbox layer in the json configuration file.

I tried adjusting rat_yolo_best.nms.json manually, but it’s clear I’m not matching what DFC expects….

  1. What is the correct way to configure NMS (rat_yolo_best.nms.json) and/or the model script (rat_yolo_best.alls) for this HAR so that optimize works?

  2. Alternatively, is there a recommended way to compile this HAR without built-in NMS, and run NMS later on the host (Raspberry Pi + Hailo-8L)?

If needed, I can provide:

rat_yolo_best.onnx
rat_yolo_best.har
rat_yolo_best.alls
rat_yolo_best.nms.json
acceleras.log
hailo_sdk.client.log

Any help getting a working .hef would mean a lot.

Thank you

2

Hey, I think you might have an issue with the end nodes. You can check these 2 threads on how you can find and specify them in your hailo parser command and .json file.

  1. The layer yolov8n/conv41 doesn’t have one output layer
  2. Yolov8s/conv41 doesn’t have one output layer

Note that in the links, they’re using the Hailo Model Zoo. However, the process with using the DFC directly is very similar.

3

Hey @Mike_Griffin,

Welcome to the Hailo Community!

So that conv43 doesn't have one output layer error you’re seeing is a pretty common issue when the YOLOv8 NMS configuration doesn’t match up with custom models.

To fix it, you’ll need to:

  • Make sure you’re using the actual end nodes and detection head layer names from your ONNX/HAR files
  • Update your rat_yolo_best.nms.json and .alls files to match those layer names

If that sounds like too much hassle, there’s another option - you could just remove the nms_postprocess from your .alls file, compile without NMS, and then handle the YOLOv8 decoding + NMS on the Raspberry Pi CPU instead.

If you want to share the relevant parts of your rat_yolo_best.nms.json and the end section of your rat_yolo_best.alls (especially the detection-related layer names), I’m happy to take a look and help you figure out which parts need to be aligned.