DFC compile problem

현호_성 · July 3, 2026, 3:07pm

I used only 64 calibration images for this first test, using EO/drone images just to verify the HEF generation flow.

The model was parsed by cutting the YOLO head before the ONNX DFL/postprocess section.

The final parser end nodes used were:

/model.23/cv2.0/cv2.0.2/Conv
/model.23/cv3.0/cv3.0.2/Conv
/model.23/cv2.1/cv2.1.2/Conv
/model.23/cv3.1/cv3.1.2/Conv
/model.23/cv2.2/cv2.2.2/Conv
/model.23/cv3.2/cv3.2.2/Conv

The HEF was successfully generated.

parse-hef result

Architecture HEF was compiled for: HAILO8
Network group name: fire_smoke_yolo11n, Multi Context - Number of contexts: 3
    Network name: fire_smoke_yolo11n/fire_smoke_yolo11n
        VStream infos:
            Input  fire_smoke_yolo11n/input_layer1 UINT8, NHWC(640x640x3)
            Output fire_smoke_yolo11n/conv51 UINT8, FCR(80x80x64)
            Output fire_smoke_yolo11n/conv54 UINT8, NHWC(80x80x2)
            Output fire_smoke_yolo11n/conv62 UINT8, FCR(40x40x64)
            Output fire_smoke_yolo11n/conv65 UINT8, NHWC(40x40x2)
            Output fire_smoke_yolo11n/conv77 UINT8, FCR(20x20x64)
            Output fire_smoke_yolo11n/conv80 UINT8, FCR(20x20x2)

So I interpret this as 3 YOLO heads:

80x80: bbox conv51, cls conv54
40x40: bbox conv62, cls conv65
20x20: bbox conv77, cls conv80

The model appears to have 2 classes.

Runtime behavior

At runtime, the model initializes correctly:

[Hailo] initialized input=fire_smoke_yolo11n/input_layer1 size=640x640 frame_size=1228800 outputs=6 classes=2

Inference speed is good, around 20~32 FPS.

However, the detections are clearly wrong. I get many boxes all over the frame, many with confidence displayed as 100%.

Example postprocess log:

[Hailo][RawYOLO] heads=3/3 selected={
80x80 box=fire_smoke_yolo11n/conv51 cls=fire_smoke_yolo11n/conv54;
40x40 box=fire_smoke_yolo11n/conv62 cls=fire_smoke_yolo11n/conv65;
20x20 box=fire_smoke_yolo11n/conv77 cls=fire_smoke_yolo11n/conv80
}
threshold=0.1
iou=0.45
scanned_cells=8400
above_threshold=1618
score_min=0
score_mean=0.171588
score_max=1
pre_nms=1617
post_nms=435
returned=100

Another frame:

above_threshold=1657
pre_nms=1652
post_nms=453
returned=100

This seems abnormal. For a YOLO model, I would not expect 1600+ cells out of 8400 to pass the confidence threshold, nor hundreds of boxes after NMS.

Tensor dump examples

For the 80x80 cls output:

cls name=fire_smoke_yolo11n/conv54
shape=80x80x2
order=NHWC
bytes=51200
floats=12800
min=-236.969
mean=-14.2728
max=43.9412

cls cell=(40,40) first_channels_NHWC=4.70799,15.6933

For 40x40 cls:

cls name=fire_smoke_yolo11n/conv65
shape=40x40x2
order=NHWC
bytes=12800
floats=3200
min=-260.11
mean=-28.5579
max=35.5706

For 20x20 cls:

cls name=fire_smoke_yolo11n/conv80
shape=20x20x2
order=FCR
bytes=3200
floats=800
min=-161.55
mean=-33.9575
max=-3.01962

The high positive class logits become almost exactly 1.0 after sigmoid, so the UI displays them as 100%.

Current raw YOLO postprocess assumption

I currently decode the raw outputs manually:

Pair bbox/cls heads by same grid size.
bbox output has 64 channels, so I assume DFL bins = 16.
cls output has 1 or 2 channels depending on model.
For each grid cell:
- apply sigmoid/logistic to cls logits
- choose best class
- decode DFL left/top/right/bottom
- generate bbox
Apply OpenCV NMS.

This worked enough to run, but the output quality is wrong.

Important question

Is this the correct way to interpret these Hailo raw YOLO outputs?

In particular:

Are the FCR bbox tensors supposed to be decoded using the same DFL layout as Ultralytics YOLO?
Are the class tensors already sigmoid-activated by DFC, or should I apply sigmoid manually?
Since parse-hef says outputs are UINT8, but my vstream buffer size equals float size, I believe HailoRT is converting output vstreams to FLOAT32. Is this correct?
Could calibration with non-fire/non-smoke images cause this type of massive over-detection, or is this more likely a postprocess decode issue?
Is there a recommended way to generate a YOLOv8/YOLO11 HEF with Hailo NMS output instead of raw 6-output tensors?

Related NMS attempt

I also tried adding Hailo NMS using a model script and JSON similar to Model Zoo YOLOv8 configs.

For custom 1-class YOLOv8-style model, the optimized HAR showed NMS metadata correctly:

NMS Meta Architecture: Yolov8
NMS Target Device: Cpu

and the HAR contained:

best_uav/yolov8_nms_postprocess

But the compiler failed at HEF build with:

BackendAllocatorException: Compilation failed: map::at

So for now I am using raw 6-output HEF and doing the YOLO postprocess myself.

What I need help with

I would like to know whether the over-detection / 100% confidence issue is caused by:

wrong raw output decoding,
wrong sigmoid / class-score handling,
wrong tensor order interpretation,
quantization / calibration issue,
or a DFC/NMS configuration issue.

Any guidance on the correct raw YOLO output layout for Hailo-compiled YOLOv8/YOLO11 models would be appreciated.

Thank you.

Jesus_Royeth · July 5, 2026, 11:38pm

Hi @user562

Your compile and parsing look fine, the problem is on the host side, in how the raw tensors are read back. There are two separate things going on and they stack up.

1. The output layout is not uniform. Look again at your own parse-hef: your six outputs do not share the same memory order. conv51, conv62 and conv77 (the 64-channel box tensors) come out as FCR, conv54 and conv65 (cls) as NHWC, and conv80 (the 20x20 cls) as FCR too. FCR is Hailo’s hardware order, it is not channel-last NHWC. On HailoRT 4.23 these are two different FormatOrder values:

from hailo_platform import FormatOrder
FormatOrder.FCR != FormatOrder.NHWC   # True

So if you take an FCR buffer and reshape it as (H, W, 64) and read the 16 DFL bins on the last axis, the values for a given cell are not contiguous and the box decode gets scrambled. That is exactly the “boxes all over the frame” symptom you see: the class scores are roughly in range but the geometry is garbage. The 20x20 head is also wrong on the cls side, because conv80 is FCR while the other two cls heads are NHWC, so a single uniform decoder misreads that one head.

The safe move is to never assume the order. Read it per output and only decode once you actually have NHWC:

for info in hef.get_output_vstream_infos():
    print(info.name, info.format.order)   # FCR vs NHWC, per output

2. Calibration. You calibrated with 64 EO/drone images that are not fire/smoke frames. That sets wrong quantization ranges, and it shows in your own dump: the cls logits go up to max=43.9, which after sigmoid is 1.000 exactly, so everything that fires reads as 100%. Recalibrate with representative frames from the real domain (a few hundred, ideally ~500–1000, with actual fire and smoke). That alone will bring the score distribution back to normal and cut the above_threshold=1618 down a lot.

On your explicit questions, quickly:

DFL layout same as Ultralytics? Yes, the math is identical (64 channels = 4 sides × 16 bins, softmax over the 16, integral, then ltrb → xyxy with the stride), but only after the tensor is NHWC. The raw FCR buffer is not channel-last.
Are the cls tensors already sigmoid-activated? No. Your end nodes cv3.x.2/Conv are the raw classification convs, before the sigmoid, and cv2.x.2/Conv is before the DFL. So you apply sigmoid yourself on cls, and the softmax+integral yourself on the box branch. Your logit range (-237 … +43) confirms these are raw logits, not probabilities.
UINT8 in parse-hef but float buffer → HailoRT converting to FLOAT32? Yes. When you request FLOAT32 output vstreams, HailoRT dequantizes UINT8 → FLOAT32 with the layer scale and zero-point on read. Your interpretation is correct.

For the recommended path: yes, compile with the on-chip NMS instead of doing raw decode. A stock Model Zoo YOLO11n exposes a single output:

Output yolov11n/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS

which gives you decoded, already-NMS’d [x1, y1, x2, y2, score] rows and skips the whole FCR/sigmoid/DFL step. Your map::at BackendAllocatorException at HEF build is a separate DFC config problem, usually the end-node order in the nms_postprocess or a classes / regression_length mismatch (should be classes=2, regression_length=16 for your yolo11n). If you paste your .alls and the nms JSON I can have a look at where the allocator fails ..

For a quick check, I would fix the calibration first and add the per-output order branch, that should turn the raw path sane; then move to on-chip NMS once the .alls is sorted.

현호_성 · July 6, 2026, 4:45am

Hi @user581,

Thanks for the previous explanation.

I would like to focus specifically on the correct way to compile my custom YOLOv8-style model into a HEF with a single Hailo NMS output.

My goal is to generate this kind of output:

Output best_uav/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS

I am not trying to debug the raw host-side decoder in this question. I want to know the correct DFC/NMS configuration needed to produce the Hailo NMS output HEF successfully.

Environment:

Target: Hailo-8
DFC versions tested: 3.33.1 and 3.34.0
Model: custom YOLOv8-style detector
Classes: 1
Input size: 640x640
Regression length: 16
NMS engine: cpu

The parser detects the YOLOv8 NMS structure and recommends these end nodes:

/model.29/cv2.0/cv2.0.2/Conv
/model.29/cv3.0/cv3.0.2/Conv
/model.29/cv2.1/cv2.1.2/Conv
/model.29/cv3.1/cv3.1.2/Conv
/model.29/cv2.2/cv2.2.2/Conv
/model.29/cv3.2/cv3.2.2/Conv

After parsing, the raw output layers are:

conv138  box, stride 8,  80x80x64
conv139  cls, stride 8,  80x80x1
conv110  box, stride 16, 40x40x64
conv111  cls, stride 16, 40x40x1
conv81   box, stride 32, 20x20x64
conv83   cls, stride 32, 20x20x1

I tried the following NMS JSON:

{
  "nms_scores_th": 0.2,
  "nms_iou_th": 0.7,
  "image_dims": [640, 640],
  "max_proposals_per_class": 100,
  "classes": 1,
  "regression_length": 16,
  "background_removal": false,
  "background_removal_index": 0,
  "bbox_decoders": [
    {
      "name": "bbox_decoder138",
      "stride": 8,
      "reg_layer": "conv138",
      "cls_layer": "conv139"
    },
    {
      "name": "bbox_decoder110",
      "stride": 16,
      "reg_layer": "conv110",
      "cls_layer": "conv111"
    },
    {
      "name": "bbox_decoder81",
      "stride": 32,
      "reg_layer": "conv81",
      "cls_layer": "conv83"
    }
  ]
}

And the ALLS file:

model_optimization_config(calibration, batch_size=8, calibset_size=1024)
post_quantization_optimization(finetune, policy=disabled)
nms_postprocess("/workspace/best_uav_nms.json", meta_arch=yolov8, engine=cpu)

The NMS command appears to be applied during optimization:

The activation function of layer best_uav/conv139 was replaced by a Sigmoid
The activation function of layer best_uav/conv111 was replaced by a Sigmoid
The activation function of layer best_uav/conv83 was replaced by a Sigmoid
Saved HAR to: /workspace/best_uav_optimized.har

Compilation then proceeds through partitioning, layer feasibility, mapping, and kernel compilation:

Partition to contexts finished successfully
Layers feasibility validated successfully
Successful Mapping
Compiling kernels of best_uav_context_0...
Compiling kernels of best_uav_context_1...
Compiling kernels of best_uav_context_2...
Compiling kernels of best_uav_context_3...
Compiling kernels of best_uav_context_4...
Building HEF...
map::at

[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: map::at

No HEF is produced.

Could you please show the correct NMS configuration for this model?

In particular, I want to know:

What should the correct bbox_decoders entries look like for these six output layers?
Should reg_layer / cls_layer use conv138 or best_uav/conv138?
Should background_removal_index be removed for a 1-class model when background_removal is false?
Is engine=cpu the correct option for producing HAILO NMS BY CLASS on Hailo-8?
Is there any additional ALLS command required to make this compile into a single NMS output HEF?

A minimal working JSON/ALLS example for this model would be very helpful.

Thank you.

Jesus_Royeth · July 6, 2026, 1:39pm

Hi

Short answer first: your recipe is basically correct, engine=cpu is the right choice, and a single HAILO NMS BY CLASS output is exactly what you should get. I reproduced the whole flow on my side in order to be sure, and I could not make your map::at happen with a standard 1-class YOLO11 — that alone tells us something. So, here’s what I found.

I compiled a custom 1-class YOLO11 (reg_max=16, so a 64-channel box head and a 1-channel cls head, same structure as yours) with the hand-written NMS path, for both hw_arch=hailo8l and hw_arch=hailo8 (your chip). Both produced the single output:

Output <net>/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS (number of classes: 1, ...)

The config that works (arch-independent, engine=cpu):

// nms.json  -- classes:1, regression_length:16 because the box head is 64 = 4*16
{
  "nms_scores_th": 0.2,
  "nms_iou_th": 0.7,
  "image_dims": [640, 640],
  "max_proposals_per_class": 100,
  "classes": 1,
  "regression_length": 16,
  "background_removal": false,
  "background_removal_index": 0,
  "bbox_decoders": [
    { "name": "bbox_decoder138", "stride": 8,  "reg_layer": "conv138", "cls_layer": "conv139" },
    { "name": "bbox_decoder110", "stride": 16, "reg_layer": "conv110", "cls_layer": "conv111" },
    { "name": "bbox_decoder81",  "stride": 32, "reg_layer": "conv81",  "cls_layer": "conv83"  }
  ]
}

# alls  -- order matters: load_model_script must run before optimize
normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
change_output_activation(conv139, sigmoid)
change_output_activation(conv111, sigmoid)
change_output_activation(conv83,  sigmoid)
nms_postprocess("./nms.json", meta_arch=yolov8, engine=cpu)

Your JSON already matches this, the config itself is not the problem. Now your six questions, each backed by a real compile:

bbox_decoders shape. Exactly 3 decoders, one per stride (8/16/32), each pairing the box conv (cv2.x) as reg_layer and the class conv (cv3.x) as cls_layer. Your mapping (conv138/conv139 @88, conv110/conv111@16@16, conv81/conv@323 @32) is correct.
conv138 vs best_uav/conv138. I tried both and they compile identically to the single NMS output, so the prefix is not your problem. Also, here’s the key part: a genuinely wrong layer name does NOT give map::at. In my tests it fails much earlier, at optimize, with HailoNNException: The layer named X doesn't exist in the HN. Your optimize succeeds (your sigmoids get replaced, the HAR saves), so your crash is not a decoder-name mismatch.
Remove background_removal_index for 1 class? No. I kept background_removal:false with background_removal_index:0 and got a clean 1-class NMS HEF. The index is inert when background_removal is false.
engine=cpu correct for Hailo-8? Yes, verified — I cross-compiled for hw_arch=hailo8 with engine=cpu and got the single HAILO NMS BY CLASS output. That is exactly how the Model Zoo HEFs are built. There is no on-chip NMS for a YOLOv8/YOLO11 DFL head, nn_core will always reject it, so engine=cpu is the intended path, not a downgrade.
Missing ALLS command? The essential set is just normalization + change_output_activation(sigmoid) on the three cls layers + nms_postprocess(json, meta_arch=yolov8, engine=cpu). Your model_optimization_config / post_quantization_optimization(finetune, policy=disabled) are fine and not the cause. I did not need allocator_param(width_splitter_defuse=disabled) in any of my compiles, so it looks optional rather than required.
Minimal working example. The two files above plus a plain ClientRunner flow: translate_onnx_model(...) → load_model_script("alls") → optimize(calib) → compile(). That produced the single-NMS HEF on both arches for me.

On the actual map::at: it happens at “Building HEF” (the allocator), after optimize and mapping already succeeded. I could not trigger it with the standard 1-class recipe on either arch — the identical valid config compiles cleanly. Your layer numbering (conv138.., /model.29/..) suggests a larger or slightly different graph than a stock yolo11n, and map::at at allocation time usually means the allocator is failing on the specific graph/partition rather than the NMS config. To pin it down I would need three things from you: your exact .alls, your nms JSON, and the full compile log (the lines just before Building HEF... map::at). One quick sanity check meanwhile: box_channels = 4 * regression_length, so a 64-channel box head needs regression_length: 16 (yours is right); if any head were 4-channel it would need 1.

Being honest about this: I run a Hailo-8L, but I did cross-compile the same recipe for hw_arch=hailo8 and it built the single NMS output there too, so the recipe is not arch-specific. Send the three files and I will look at where the allocator trips.

현호_성 · July 7, 2026, 4:43am

Hi, thank you very much for taking the time to reproduce it on both Hailo-8L and Hailo-8. That is very helpful.

Your results are actually encouraging because they suggest that the NMS JSON and ALLS configuration are fundamentally correct, and that my map::at issue is probably related to my specific graph rather than the NMS configuration itself.

At this point, I think I may be focusing too much on debugging my current project. Instead, I’d like to understand the recommended workflow for converting a custom YOLO ONNX model into a single HAILO NMS BY CLASS HEF.

Could you describe the complete “official” workflow that you recommend?

For example:

Which ONNX nodes should be used as the parser end nodes?
Should I always stop at the six final Conv layers (cv2/cv3), or is there another recommended point?
At what stage should change_output_activation() and nms_postprocess() be applied?
Are they always loaded through the model script before optimize(), or are there cases where they should be applied differently?
Is the recommended pipeline essentially:

ONNX
    ↓
parser (last Conv layers)
    ↓
load_model_script(.alls)
    ↓
optimize(calibration)
    ↓
compiler
    ↓
Single HAILO NMS BY CLASS output

or is there any additional recommended step?

Does this workflow also apply to DFC 3.34, or has anything changed compared to 3.33?

I would like to build my future projects following the same workflow that the Model Zoo uses, rather than relying on custom raw tensor decoders.

If there is any documentation or an official example showing the recommended conversion process for a custom YOLO11 model with NMS output, I would really appreciate it.

Thank you again for reproducing the issue and for all the detailed explanations.

현호_성 · July 7, 2026, 4:55am

Hi,

I have an update.

I was able to successfully compile the YOLO11n model with Hailo NMS integrated into the HEF. The generated HEF contains a single HAILO NMS BY CLASS output, so the NMS integration is now working correctly.

This makes me think the previous map::at error was likely related to the larger model (or the graph/resource allocation during HEF generation), rather than my NMS JSON or .alls configuration.

However, I’m now seeing another issue during inference. I’ve attached a screenshot.

The HEF is running with the integrated Hailo NMS, but the detection results are clearly incorrect:

Many false positives appear in unrelated areas.
Some confidence scores are very close to or fixed at 100%.
Bounding boxes are scattered and do not correspond to the actual fire regions.

For comparison, when I run the same ONNX model directly (without Hailo), the detections look normal.

Do you have any idea what could cause this behavior? Could this be related to quantization/calibration, output tensor parsing, NMS decoding, or something else?

I’ve attached the screenshot for reference. Any suggestions would be greatly appreciated.

Jesus_Royeth · July 7, 2026, 3:09pm

Hi @user562, good to see the single HAILO NMS BY CLASS HEF compiling now. Let me follow up on the bad inference, because I think that part has a different cause than the compile side.

First of all: I would not blame double normalisation here. I ran a small A/B test on my own Hailo-8L (HailoRT 4.23) with an equivalent 1-class YOLO11n HEF — same HAILO NMS BY CLASS output (engine=cpu), the same baked normalization([0,0,0],[255,255,255]), score threshold 0.2 in the HEF. When I fed the frame already divided by 255 (the pre-process you inherit from the ONNX side), the chip divides again, the network sees an almost black image, and the output was empty: 0 detections on all frames, not many boxes. With the correct uint8 0-255 input I got 11-13 clean detections per frame, scores 0.21-0.92. So a baked-in second /255 tends to give you an empty output, not many boxes. This is on my own model, not your fire one, but the mechanism is the same, so your symptom (many scattered boxes, confidences near 100%) does not look like double normalisation to me.

What the three symptoms together point at, in my opinion, is that the host is still running your old raw decoder on top of an output that is already final. With the NMS inside the HEF, yolov8_nms_postprocess returns the final result in a single vstream: boxes already DFL-decoded, class scores already passed through sigmoid, NMS already applied, and the layout is decoded rows grouped by class (with a per-class count), not the six grid tensors your decoder was written for. If your host code from the first post still reads that buffer as raw heads and runs sigmoid + DFL + OpenCV NMS on it, it is feeding the wrong bytes — normalised coordinates and per-class counts — into the sigmoid and the DFL integral, so the boxes end up scattered and the scores saturate. That is the first thing I would check: it explains the three symptoms, and it also explains why the plain ONNX looks fine, because there the decoder runs against the raw heads it was written for.

In order to confirm it, the check is simple: dump the NMS output rows raw, with no post-processing of your own. They should already come as decoded detections per class — four normalised box coordinates plus a score in [0,1] (multiply the coords by 640 to draw). One detail about the coordinate order: Hailo returns the NMS boxes as y_min, x_min, y_max, x_max, not x_min first, so a naive read looks wrong even when the data is fine. If the rows look sane, you can delete the whole DFL/sigmoid/NMS path on the host and just draw them, that code is dead now.

Two smaller things to rule out, in order:

Input dtype/range going into HailoRT: print dtype/min/max of the buffer right before you write the input vstream. It must be uint8 in [0,255], not float in [0,1], because the normalisation is baked into the HEF. It takes 30 seconds to check, but note that in my test the wrong input gave an empty output, not your current symptom, so I would not expect this alone to explain what you see.
Calibration: those 64 frames “just to verify the flow” are too few and not representative, and that already showed up as saturated logits in your earlier dump. It is a real factor, but on its own it tends to shift the scores, not to put boxes all over the frame, and “ONNX good / Hailo bad” points more at the host parse than at quantisation. I did not test this part — I can’t recompile with your fire/smoke set — so I would fix the decoder path first and only recalibrate if the boxes are still wrong after that.

I can’t see your graph or your read-back code, so take this as a hypothesis, not a verdict. If you paste the snippet of how you read the HEF output and turn it into boxes, it should be quick to confirm which one it is ..