Compilation fails on Hailo-8L: "16x4 is not supported" kernel validation error on stride-2 conv (custom Whisper-small encoder)

General Discussion
1

Converting a custom-exported Whisper small encoder for Hailo-8L (DFC 3.33.1). Optimization completes successfully (full calibration, 4-epoch QAT finetune, bias correction, adaround). Compilation fails on one layer.

Failing layer (conv2) — the stride-2 downsampling conv in Whisper’s input stem:

kernel_shape: [1, 3, 768, 768]
strides: [1, 1, 2, 1]
input_shape: [-1, 1, 500, 768]
output_shape: [-1, 1, 250, 768]

Error

[error] Mapping Failed
conv2 failed on kernel validation: 16x4 is not supported in conv2
conv2: Minimum needed amount of subclusters was not reached. Needed: 4, but requests only 1/2/3 subclusters

Automatic decomposition (conv2_d0…conv2_d36+) is attempted but every sub-piece fails the same kernel validation. Adding quantization_param(conv2, precision_mode=a16_w16) had no effect (makes sense — looks structural, not precision-related).

This same layer works fine on Whisper tiny/base at lower channel counts (384/512); small’s 768 channels seems to push this stride-2 kernel shape past whatever subcluster/tiling limit Hailo-8L has.

Is this kernel_shape/stride combo a known hard limit on Hailo-8L, or is there a .alls directive (conv splitting/decomposition) to make it map successfully?

2

Hi @user562 :waving_hand:

Your compile and parsing look fine, the problem is on the host side, in how the raw tensors are read back. There are two separate things going on and they stack up.

1. The output layout is not uniform. Look again at your own parse-hef: your six outputs do not share the same memory order. conv51, conv62 and conv77 (the 64-channel box tensors) come out as FCR, conv54 and conv65 (cls) as NHWC, and conv80 (the 20x20 cls) as FCR too. FCR is Hailo’s hardware order, it is not channel-last NHWC. On HailoRT 4.23 these are two different FormatOrder values:

from hailo_platform import FormatOrder
FormatOrder.FCR != FormatOrder.NHWC   # True

So if you take an FCR buffer and reshape it as (H, W, 64) and read the 16 DFL bins on the last axis, the values for a given cell are not contiguous and the box decode gets scrambled. That is exactly the “boxes all over the frame” symptom you see: the class scores are roughly in range but the geometry is garbage. The 20x20 head is also wrong on the cls side, because conv80 is FCR while the other two cls heads are NHWC, so a single uniform decoder misreads that one head.

The safe move is to never assume the order. Read it per output and only decode once you actually have NHWC:

for info in hef.get_output_vstream_infos():
    print(info.name, info.format.order)   # FCR vs NHWC, per output

2. Calibration. You calibrated with 64 EO/drone images that are not fire/smoke frames. That sets wrong quantization ranges, and it shows in your own dump: the cls logits go up to max=43.9, which after sigmoid is 1.000 exactly, so everything that fires reads as 100%. Recalibrate with representative frames from the real domain (a few hundred, ideally ~500–1000, with actual fire and smoke). That alone will bring the score distribution back to normal and cut the above_threshold=1618 down a lot.

On your explicit questions, quickly:

  • DFL layout same as Ultralytics? Yes, the math is identical (64 channels = 4 sides × 16 bins, softmax over the 16, integral, then ltrb → xyxy with the stride), but only after the tensor is NHWC. The raw FCR buffer is not channel-last.
  • Are the cls tensors already sigmoid-activated? No. Your end nodes cv3.x.2/Conv are the raw classification convs, before the sigmoid, and cv2.x.2/Conv is before the DFL. So you apply sigmoid yourself on cls, and the softmax+integral yourself on the box branch. Your logit range (-237 … +43) confirms these are raw logits, not probabilities.
  • UINT8 in parse-hef but float buffer → HailoRT converting to FLOAT32? Yes. When you request FLOAT32 output vstreams, HailoRT dequantizes UINT8 → FLOAT32 with the layer scale and zero-point on read. Your interpretation is correct.

For the recommended path: yes, compile with the on-chip NMS instead of doing raw decode. A stock Model Zoo YOLO11n exposes a single output:

Output yolov11n/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS

which gives you decoded, already-NMS’d [x1, y1, x2, y2, score] rows and skips the whole FCR/sigmoid/DFL step. Your map::at BackendAllocatorException at HEF build is a separate DFC config problem, usually the end-node order in the nms_postprocess or a classes / regression_length mismatch (should be classes=2, regression_length=16 for your yolo11n). If you paste your .alls and the nms JSON I can have a look at where the allocator fails ..

For a quick check, I would fix the calibration first and add the per-output order branch, that should turn the raw path sane; then move to on-chip NMS once the .alls is sorted.

3

Thanks for your detailed explanation. It makes sense, especially regarding the FCR vs NHWC layout.

However, I would like to add one important observation.

The over-detection issue is not limited to the fire/smoke model. I originally experienced exactly the same behavior with my UAV model, which was compiled from a UAV ONNX model and calibrated using representative UAV images. Therefore, I don’t think the calibration dataset is the primary cause of this issue. The fire/smoke experiment was only an additional test to check whether the behavior was model-specific, but the same symptoms appeared.

At this point, I believe the root cause is more likely related to how the raw outputs are interpreted on the host side.

I understand that I need to handle the output layout correctly (FCR vs NHWC) for each output tensor. However, implementing and maintaining a custom decoder for every possible output layout is becoming quite difficult, especially since different models expose different combinations of FCR and NHWC outputs.

Because of that, I would like to move to the Hailo NMS output path instead of decoding the raw tensors myself.

Unfortunately, I have not been able to generate a HEF with a single NMS output.

My current environment is:

  • DFC: 3.33.1

  • HailoRT: 4.23.0

  • Hardware: Hailo-8

I can successfully compile the model, but the generated HEF still exposes the six raw outputs:

  • conv51

  • conv54

  • conv62

  • conv65

  • conv77

  • conv80

instead of a single HAILO NMS BY CLASS output like the official Model Zoo YOLOv8/YOLO11 HEFs.

I also tried using:



nms_postprocess(..., meta_arch=yolov8, engine=nn_core)

but DFC reports:

The specified meta architecture yolov8 cannot be run on chip.

Using engine=cpu compiles successfully, but the generated HEF still exposes the six raw outputs instead of a single NMS output.

Could this be a limitation or a known issue in DFC 3.33.1?

Has this behavior been changed or fixed in a newer DFC release?

Could you please explain the recommended way to generate a single HAILO NMS output (similar to the official Model Zoo YOLOv8/YOLO11 HEFs) for a custom YOLO11 model?

I would really appreciate any guidance or example workflow. It would greatly simplify the host-side implementation and eliminate the need to maintain custom decoders for different tensor layouts.

Thank you again for your help.

4

Hi @user583,

Thank you for your detailed explanation regarding the raw decoder. I agree that my over-detection issue is most likely on the host side (FCR/NHWC handling), and I will continue debugging that separately.

However, I also investigated the on-chip NMS path that you recommended, and I am still unable to generate an NMS HEF.

I would like to provide an update because this seems to be a different issue from the raw decoder.

Environment

  • Hailo-8 (not Hailo-8L)

  • DFC 3.33.1

  • DFC 3.34.0 (fresh virtual environment, clean installation)

  • YOLOv8-based custom model

  • 1 class

  • regression_length = 16

  • meta_arch = yolov8

  • engine = cpu

Model

Parser detects the model correctly:



NMS structure of yolov8 (or equivalent architecture) was detected.

In order to use HailoRT post-processing capabilities, these end node names should be used:

/model.29/cv2.0/cv2.0.2/Conv
/model.29/cv3.0/cv3.0.2/Conv
/model.29/cv2.1/cv2.1.2/Conv
/model.29/cv3.1/cv3.1.2/Conv
/model.29/cv2.2/cv2.2.2/Conv
/model.29/cv3.2/cv3.2.2/Conv

I used exactly those end nodes.

My NMS JSON is:



{
  "nms_scores_th": 0.2,
  "nms_iou_th": 0.7,
  "image_dims": [640,640],
  "max_proposals_per_class": 100,
  "classes": 1,
  "regression_length": 16,
  "background_removal": false,
  "background_removal_index": 0,
  "bbox_decoders":[
    {"name":"bbox_decoder138","stride":8,"reg_layer":"conv138","cls_layer":"conv139"},
    {"name":"bbox_decoder110","stride":16,"reg_layer":"conv110","cls_layer":"conv111"},
    {"name":"bbox_decoder81","stride":32,"reg_layer":"conv81","cls_layer":"conv83"}
  ]
}

and my model script is simply



model_optimization_config(calibration, batch_size=8, calibset_size=1024)
post_quantization_optimization(finetune, policy=disabled)
nms_postprocess("/workspace/best_uav_nms.json", meta_arch=yolov8, engine=cpu)

What succeeds

Everything succeeds until the final HEF generation.

  • parser ✓

  • optimize ✓

  • calibration ✓

  • optimized HAR ✓

  • partition ✓

  • mapping ✓

  • kernel compilation ✓

For example:



Successful Mapping (allocation time: 23m 55s)

Compiling kernels...

Bandwidth calculation...

Building HEF...

Failure

The compiler fails only during the final HEF generation:



Building HEF...
map::at

BackendAllocatorException:
Compilation failed: map::at

No HEF is produced.

Same behavior on DFC 3.33.1 and 3.34.0

Originally I thought this might be fixed in DFC 3.34.0.

However, after creating a completely fresh Python virtual environment and reinstalling DFC 3.34.0 from scratch, I get exactly the same failure.

The compilation proceeds much further than before (mapping and kernel compilation complete successfully), but it still fails at the final “Building HEF…” stage with the same map::at exception.

My question

Does this look like a known DFC backend issue?

Or is there something incorrect in my NMS JSON / ALLS configuration that could still trigger this allocator failure?

Since parser automatically detected the YOLOv8 NMS structure and I am using the parser-recommended end nodes with classes=1 and regression_length=16, I am running out of ideas.

If there are any additional debug flags, compiler logs, or HAR files that would help identify where map::at is thrown, I would be happy to provide them.

Thank you!

5

DFC 3.34.0, Hailo-8, custom YOLOv8 1-class.

Raw HEF compiles successfully.

With nms_postprocess(engine=cpu), NMS graph is generated correctly:

  • yolov8_nms_postprocess exists
  • postprocess_output_layer exists
  • output shape is [-1, 1, 5, 100]
  • engine is cpu
  • inputs are conv138, conv139, conv110, ne_activation_conv111, conv81, conv83

But compiler crashes after partition:
Applying selected partition to 3 contexts…
BackendAllocatorException: Compilation failed with unexpected crash

Without explicit change_output_activation, it reaches Building HEF and fails with:
map::at

engine=auto fails earlier:
UnsupportedMetaArchError: yolov8 cannot be run on chip

What is the correct supported DFC 3.34 configuration for custom YOLOv8 1-class Hailo NMS BY CLASS HEF?