Custom Yolov11 not detecting after compilation

General
,
1

Hello,
I’ve compiled my ONNX file using the Hailo Dataflow Compiler. The ONNX file was retrained with custom data based on YOLOv11s. After testing the model on my Hailo, the performance is very poor — it can’t detect most of the objects. Note that I tested the original .pt model, and it performs well. I also previously compiled a custom YOLOv8 model, and it worked fine. Additionally, I have already checked the output node names for YOLOv11, and the compilation didn’t raise any errors.

Here is the .alls file

normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
change_output_activation(conv54, sigmoid)
change_output_activation(conv65, sigmoid)
change_output_activation(conv80, sigmoid)
model_optimization_flavor(optimization_level=4)
nms_postprocess("nms_layer_config.json", meta_arch=yolov8, engine=cpu)

and the .json:

{
	"nms_scores_th": 0.3,
	"nms_iou_th": 0.7,
	"image_dims": [
		640,
		640
	],
	"max_proposals_per_class": 100,
	"classes": 1,
	"regression_length": 16,
	"background_removal": false,
	"background_removal_index": 0,
	"bbox_decoders": [
		{
			"name": "bbox_decoder51",
			"stride": 8,
			"reg_layer": "conv51",
			"cls_layer": "conv54"
		},
		{
			"name": "bbox_decoder62",
			"stride": 16,
			"reg_layer": "conv62",
			"cls_layer": "conv65"
		},
		{
			"name": "bbox_decoder77",
			"stride": 32,
			"reg_layer": "conv77",
			"cls_layer": "conv80"
		}
	]
}

I also lowered the optimization level to 2, but nothing changed.

Has anyone else had issues with YOLOv11? I’m very confused. I don’t know what could be wrong. I followed the instructions from this guide: Guide to using the DFC to convert a modified YoloV11 on Google Colab and also recompiled using the CLI commands instead.

Thanks!

2

Hey @dgarrido ,

Here are some things you can try to get better accuracy and performance :

1. Reduce Optimization Level

model_optimization_flavor(optimization_level=2)

Level 4 is too aggressive and likely causing accuracy loss. Start with level 2.

  • Optimization Level 4 enables Equalization + Adaround (320 epochs & 1024 images on all layers).
  • This level is highly aggressive and might over-optimize, causing a loss in model accuracy.

2. Review Output Activations
The manual sigmoid activations may be redundant:

change_output_activation(conv54, sigmoid)
change_output_activation(conv65, sigmoid)
change_output_activation(conv80, sigmoid)

YOLO models typically apply sigmoid inside their detection heads, meaning manually adding these activations may cause incorrect outputs.

3. Verify Bounding Box Decoder Layers
Ensure your bbox_decoder configuration correctly maps to YOLOv11’s detection heads:

  • Check if conv54, conv65, and conv80 are proper classification layers
  • Verify reg_layer and cls_layer assignments match YOLOv11’s structure

4. Adjust NMS Thresholds

"nms_scores_th": 0.2,
"nms_iou_th": 0.5

Current settings (0.3/0.7) might be causing detection issues.

3

Thanks, Omria! I’m going to try this. I was using this .alls file from the model zoo as a reference: hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov11s.alls at master · hailo-ai/hailo_model_zoo.

I also noticed that I was using images with padding in the calibration set to maintain the aspect ratio. Then, I tried images with only resizing, without padding, and the performance is much better.

1 Like