DFC compile problem

user562 · July 3, 2026, 3:07pm

I used only 64 calibration images for this first test, using EO/drone images just to verify the HEF generation flow.

The model was parsed by cutting the YOLO head before the ONNX DFL/postprocess section.

The final parser end nodes used were:

/model.23/cv2.0/cv2.0.2/Conv
/model.23/cv3.0/cv3.0.2/Conv
/model.23/cv2.1/cv2.1.2/Conv
/model.23/cv3.1/cv3.1.2/Conv
/model.23/cv2.2/cv2.2.2/Conv
/model.23/cv3.2/cv3.2.2/Conv

The HEF was successfully generated.

parse-hef result

Architecture HEF was compiled for: HAILO8
Network group name: fire_smoke_yolo11n, Multi Context - Number of contexts: 3
    Network name: fire_smoke_yolo11n/fire_smoke_yolo11n
        VStream infos:
            Input  fire_smoke_yolo11n/input_layer1 UINT8, NHWC(640x640x3)
            Output fire_smoke_yolo11n/conv51 UINT8, FCR(80x80x64)
            Output fire_smoke_yolo11n/conv54 UINT8, NHWC(80x80x2)
            Output fire_smoke_yolo11n/conv62 UINT8, FCR(40x40x64)
            Output fire_smoke_yolo11n/conv65 UINT8, NHWC(40x40x2)
            Output fire_smoke_yolo11n/conv77 UINT8, FCR(20x20x64)
            Output fire_smoke_yolo11n/conv80 UINT8, FCR(20x20x2)

So I interpret this as 3 YOLO heads:

80x80: bbox conv51, cls conv54
40x40: bbox conv62, cls conv65
20x20: bbox conv77, cls conv80

The model appears to have 2 classes.

Runtime behavior

At runtime, the model initializes correctly:

[Hailo] initialized input=fire_smoke_yolo11n/input_layer1 size=640x640 frame_size=1228800 outputs=6 classes=2

Inference speed is good, around 20~32 FPS.

However, the detections are clearly wrong. I get many boxes all over the frame, many with confidence displayed as 100%.

Example postprocess log:

[Hailo][RawYOLO] heads=3/3 selected={
80x80 box=fire_smoke_yolo11n/conv51 cls=fire_smoke_yolo11n/conv54;
40x40 box=fire_smoke_yolo11n/conv62 cls=fire_smoke_yolo11n/conv65;
20x20 box=fire_smoke_yolo11n/conv77 cls=fire_smoke_yolo11n/conv80
}
threshold=0.1
iou=0.45
scanned_cells=8400
above_threshold=1618
score_min=0
score_mean=0.171588
score_max=1
pre_nms=1617
post_nms=435
returned=100

Another frame:

above_threshold=1657
pre_nms=1652
post_nms=453
returned=100

This seems abnormal. For a YOLO model, I would not expect 1600+ cells out of 8400 to pass the confidence threshold, nor hundreds of boxes after NMS.

Tensor dump examples

For the 80x80 cls output:

cls name=fire_smoke_yolo11n/conv54
shape=80x80x2
order=NHWC
bytes=51200
floats=12800
min=-236.969
mean=-14.2728
max=43.9412

cls cell=(40,40) first_channels_NHWC=4.70799,15.6933

For 40x40 cls:

cls name=fire_smoke_yolo11n/conv65
shape=40x40x2
order=NHWC
bytes=12800
floats=3200
min=-260.11
mean=-28.5579
max=35.5706

For 20x20 cls:

cls name=fire_smoke_yolo11n/conv80
shape=20x20x2
order=FCR
bytes=3200
floats=800
min=-161.55
mean=-33.9575
max=-3.01962

The high positive class logits become almost exactly 1.0 after sigmoid, so the UI displays them as 100%.

Current raw YOLO postprocess assumption

I currently decode the raw outputs manually:

Pair bbox/cls heads by same grid size.
bbox output has 64 channels, so I assume DFL bins = 16.
cls output has 1 or 2 channels depending on model.
For each grid cell:
- apply sigmoid/logistic to cls logits
- choose best class
- decode DFL left/top/right/bottom
- generate bbox
Apply OpenCV NMS.

This worked enough to run, but the output quality is wrong.

Important question

Is this the correct way to interpret these Hailo raw YOLO outputs?

In particular:

Are the FCR bbox tensors supposed to be decoded using the same DFL layout as Ultralytics YOLO?
Are the class tensors already sigmoid-activated by DFC, or should I apply sigmoid manually?
Since parse-hef says outputs are UINT8, but my vstream buffer size equals float size, I believe HailoRT is converting output vstreams to FLOAT32. Is this correct?
Could calibration with non-fire/non-smoke images cause this type of massive over-detection, or is this more likely a postprocess decode issue?
Is there a recommended way to generate a YOLOv8/YOLO11 HEF with Hailo NMS output instead of raw 6-output tensors?

Related NMS attempt

I also tried adding Hailo NMS using a model script and JSON similar to Model Zoo YOLOv8 configs.

For custom 1-class YOLOv8-style model, the optimized HAR showed NMS metadata correctly:

NMS Meta Architecture: Yolov8
NMS Target Device: Cpu

and the HAR contained:

best_uav/yolov8_nms_postprocess

But the compiler failed at HEF build with:

BackendAllocatorException: Compilation failed: map::at

So for now I am using raw 6-output HEF and doing the YOLO postprocess myself.

What I need help with

I would like to know whether the over-detection / 100% confidence issue is caused by:

wrong raw output decoding,
wrong sigmoid / class-score handling,
wrong tensor order interpretation,
quantization / calibration issue,
or a DFC/NMS configuration issue.

Any guidance on the correct raw YOLO output layout for Hailo-compiled YOLOv8/YOLO11 models would be appreciated.

Thank you.

user581 · July 5, 2026, 11:38pm

Hi @user562

Your compile and parsing look fine, the problem is on the host side, in how the raw tensors are read back. There are two separate things going on and they stack up.

1. The output layout is not uniform. Look again at your own parse-hef: your six outputs do not share the same memory order. conv51, conv62 and conv77 (the 64-channel box tensors) come out as FCR, conv54 and conv65 (cls) as NHWC, and conv80 (the 20x20 cls) as FCR too. FCR is Hailo’s hardware order, it is not channel-last NHWC. On HailoRT 4.23 these are two different FormatOrder values:

from hailo_platform import FormatOrder
FormatOrder.FCR != FormatOrder.NHWC   # True

So if you take an FCR buffer and reshape it as (H, W, 64) and read the 16 DFL bins on the last axis, the values for a given cell are not contiguous and the box decode gets scrambled. That is exactly the “boxes all over the frame” symptom you see: the class scores are roughly in range but the geometry is garbage. The 20x20 head is also wrong on the cls side, because conv80 is FCR while the other two cls heads are NHWC, so a single uniform decoder misreads that one head.

The safe move is to never assume the order. Read it per output and only decode once you actually have NHWC:

for info in hef.get_output_vstream_infos():
    print(info.name, info.format.order)   # FCR vs NHWC, per output

2. Calibration. You calibrated with 64 EO/drone images that are not fire/smoke frames. That sets wrong quantization ranges, and it shows in your own dump: the cls logits go up to max=43.9, which after sigmoid is 1.000 exactly, so everything that fires reads as 100%. Recalibrate with representative frames from the real domain (a few hundred, ideally ~500–1000, with actual fire and smoke). That alone will bring the score distribution back to normal and cut the above_threshold=1618 down a lot.

On your explicit questions, quickly:

DFL layout same as Ultralytics? Yes, the math is identical (64 channels = 4 sides × 16 bins, softmax over the 16, integral, then ltrb → xyxy with the stride), but only after the tensor is NHWC. The raw FCR buffer is not channel-last.
Are the cls tensors already sigmoid-activated? No. Your end nodes cv3.x.2/Conv are the raw classification convs, before the sigmoid, and cv2.x.2/Conv is before the DFL. So you apply sigmoid yourself on cls, and the softmax+integral yourself on the box branch. Your logit range (-237 … +43) confirms these are raw logits, not probabilities.
UINT8 in parse-hef but float buffer → HailoRT converting to FLOAT32? Yes. When you request FLOAT32 output vstreams, HailoRT dequantizes UINT8 → FLOAT32 with the layer scale and zero-point on read. Your interpretation is correct.

For the recommended path: yes, compile with the on-chip NMS instead of doing raw decode. A stock Model Zoo YOLO11n exposes a single output:

Output yolov11n/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS

which gives you decoded, already-NMS’d [x1, y1, x2, y2, score] rows and skips the whole FCR/sigmoid/DFL step. Your map::at BackendAllocatorException at HEF build is a separate DFC config problem, usually the end-node order in the nms_postprocess or a classes / regression_length mismatch (should be classes=2, regression_length=16 for your yolo11n). If you paste your .alls and the nms JSON I can have a look at where the allocator fails ..

For a quick check, I would fix the calibration first and add the per-output order branch, that should turn the raw path sane; then move to on-chip NMS once the .alls is sorted.

user562 · July 6, 2026, 4:43am

(post deleted by author)

user562 · July 6, 2026, 4:45am

Hi @user581,

Thanks for the previous explanation.

I would like to focus specifically on the correct way to compile my custom YOLOv8-style model into a HEF with a single Hailo NMS output.

My goal is to generate this kind of output:

Output best_uav/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS

I am not trying to debug the raw host-side decoder in this question. I want to know the correct DFC/NMS configuration needed to produce the Hailo NMS output HEF successfully.

Environment:

Target: Hailo-8
DFC versions tested: 3.33.1 and 3.34.0
Model: custom YOLOv8-style detector
Classes: 1
Input size: 640x640
Regression length: 16
NMS engine: cpu

The parser detects the YOLOv8 NMS structure and recommends these end nodes:

/model.29/cv2.0/cv2.0.2/Conv
/model.29/cv3.0/cv3.0.2/Conv
/model.29/cv2.1/cv2.1.2/Conv
/model.29/cv3.1/cv3.1.2/Conv
/model.29/cv2.2/cv2.2.2/Conv
/model.29/cv3.2/cv3.2.2/Conv

After parsing, the raw output layers are:

conv138  box, stride 8,  80x80x64
conv139  cls, stride 8,  80x80x1
conv110  box, stride 16, 40x40x64
conv111  cls, stride 16, 40x40x1
conv81   box, stride 32, 20x20x64
conv83   cls, stride 32, 20x20x1

I tried the following NMS JSON:

{
  "nms_scores_th": 0.2,
  "nms_iou_th": 0.7,
  "image_dims": [640, 640],
  "max_proposals_per_class": 100,
  "classes": 1,
  "regression_length": 16,
  "background_removal": false,
  "background_removal_index": 0,
  "bbox_decoders": [
    {
      "name": "bbox_decoder138",
      "stride": 8,
      "reg_layer": "conv138",
      "cls_layer": "conv139"
    },
    {
      "name": "bbox_decoder110",
      "stride": 16,
      "reg_layer": "conv110",
      "cls_layer": "conv111"
    },
    {
      "name": "bbox_decoder81",
      "stride": 32,
      "reg_layer": "conv81",
      "cls_layer": "conv83"
    }
  ]
}

And the ALLS file:

model_optimization_config(calibration, batch_size=8, calibset_size=1024)
post_quantization_optimization(finetune, policy=disabled)
nms_postprocess("/workspace/best_uav_nms.json", meta_arch=yolov8, engine=cpu)

The NMS command appears to be applied during optimization:

The activation function of layer best_uav/conv139 was replaced by a Sigmoid
The activation function of layer best_uav/conv111 was replaced by a Sigmoid
The activation function of layer best_uav/conv83 was replaced by a Sigmoid
Saved HAR to: /workspace/best_uav_optimized.har

Compilation then proceeds through partitioning, layer feasibility, mapping, and kernel compilation:

Partition to contexts finished successfully
Layers feasibility validated successfully
Successful Mapping
Compiling kernels of best_uav_context_0...
Compiling kernels of best_uav_context_1...
Compiling kernels of best_uav_context_2...
Compiling kernels of best_uav_context_3...
Compiling kernels of best_uav_context_4...
Building HEF...
map::at

[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: map::at

No HEF is produced.

Could you please show the correct NMS configuration for this model?

In particular, I want to know:

What should the correct bbox_decoders entries look like for these six output layers?
Should reg_layer / cls_layer use conv138 or best_uav/conv138?
Should background_removal_index be removed for a 1-class model when background_removal is false?
Is engine=cpu the correct option for producing HAILO NMS BY CLASS on Hailo-8?
Is there any additional ALLS command required to make this compile into a single NMS output HEF?

A minimal working JSON/ALLS example for this model would be very helpful.

Thank you.