DFC 5.3.0 / Hailo-10H: YOLOv8s fails with Value doesn't fit in field (1951, 119) at 4064×3040, and with Agent infeasible when on-chip NV12/I420 input is combined with compiler_optimization_level=max

General Discussion
, ,
1

Hi all,

we compile a YOLOv8s detector (2 classes, raw head outputs, no on-chip NMS) at a native sensor resolution of 4064×3040 for Hailo-10H. It compiles with compiler_optimization_level=0, but every run at level 1 aborts with BackendAllocatorException: Value doesn't fit in field (1951). That happens after the partitioner has already found and logged valid partitions. The abort is deterministic: the same candidate sequence and the same value every time. At level max the search completes (valid 6-context partition, 9.56 FPS estimate), and the compile then fails in the next stage with field 119.

A second, separate failure: at 3008×2240, the same model compiles at level max with RGB input, but every build with on-chip YUV input conversion (NV12 or I420) fails at level max with Validator failed on node: conv21 with Agent infeasible. The same YUV builds compile fine at level 0. Details in part 2 below.

Environment

  • DFC 5.3.0 (hailo_sdk_client 5.3.0), Docker image with CUDA 12.5.1, RTX A6000, Ubuntu 24.04 host.

  • Target hailo10h; runtime HailoRT 5.3.0 on a Raspberry Pi 5.

Model and script

  • YOLOv8s (Ultralytics 8.4.42), ONNX opset 11, static input 1×3×3040×4064.

  • Parsed with end_node_names = the six head convolutions (/model.22/cv2.{0,1,2}/…/Conv, /model.22/cv3.{0,1,2}/…/Conv).

  • Optimize script:

  normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
  model_optimization_flavor(optimization_level=0, compression_level=0, batch_size=1)
  model_optimization_config(calibration, batch_size=1, calibset_size=64)
  quantization_param([conv63], precision_mode=a16_w16)
  • Compile: the optimized HAR is loaded, a compile-only script is loaded with load_model_script(...), then runner.compile().

What happens

Compile script Result
performance_param(compiler_optimization_level=0) OK: 6 contexts, 19 min. Runs on Hailo-10H at 2.06 FPS, matching the partitioner’s estimate of 2.10.
performance_param(compiler_optimization_level=1) Abort after 18 min
=1 + context_switch_param(mode=enabled) Abort after 18 min. Identical candidate sequence and identical abort point.
=1 + resources_param(max_compute_utilization=0.5, max_control_utilization=0.5, max_memory_utilization=0.5) Abort after 35 min. The best estimate reached was 4.63 FPS.
performance_param(compiler_optimization_level=max) (logged as optimization_level(max=2)) Fail after 1 h 58 min. Successful Multi Context Partition (6 contexts, Pro Unikorn 9.56 FPS) and Partition passed at control_util 1.0, then Partition and Allocation Failed → Value doesn't fit in field (119)
performance_param(fps=6) Abort after 18 min, field 1951 (the FPS-target flow seems to follow the level-1 path)
=max with NV12 input (input_conversion(input_layer1, nv12_to_rgb) in the optimize script) Abort after 1 h 38 min, field 1951
=max + resources_param(...) Rejected up front: Performance Flow requires automatic resource utilization

The same model and script compile without problems at 1280×960 (=1, 5 contexts), 2560×1920 and 3008×2240 (=0, 3 contexts).

Log (hailo_sdk.core.log, =1 run, trimmed):

Single context flow failed: Recoverable single context error - Resources presolve failed: lcus=(141/80)
Using Multi-context flow
Resources optimization params: max_control_utilization=60%, max_compute_utilization=60%, ...
...
Iteration failed on: Number of rows buffered from reshape_from_conv_feature_splitter2_1_to_concat2 to concat2 (257) exceeds maximal (255)
...
Found valid partition to 6 contexts                                   (iteration #102, Unikorn FPS 3.12)
Iteration failed on: Automri finished with too many resources on context_4
Found valid partition to 6 contexts, Performance improved by 48.5%    (iteration #104, Unikorn FPS 3.22)
Iteration failed on: Automri finished with too many resources on context_1
Partition iteration #106:  Fast Unikorn FPS: 3.46491
  context_0..5 Status: VALID
Partition failed on: Value doesn't fit in field (1951)
yolov8s_det Multi Context Partition Failed (duration: 16m 34s)
Builder fail with status 101: Value doesn't fit in field (1951)
BackendAllocatorException: Compilation failed: Value doesn't fit in field (1951)

Log (=max run, end):

Partition failed on: Pro unikorn is below the best. Current: 8.642794, Best: 9.556581
...
Applying selected partition to 6 contexts...
context_0..5 - Bottleneck FPS: 47.33, 38.09, 57.29, 47.97, 206.21, 303.94
yolov8s_det Successful Multi Context Partition (duration: 1h 56m 1s)
Partition passed at control_util 1.0.
Memory measurement: ... Peak RSS: 13662146560 bytes, stage: sloth
yolov8s_det Partition and Allocation Failed (duration: 1h 57m 43s)
Builder fail with status 101: Value doesn't fit in field (119)

Part 2: Agent infeasible with on-chip YUV input at level max (3008×2240)

Same model, recipe and DFC. Input 3008×2240; the only difference between builds is the input format, added to the optimize script as

nv12_to_yuv1, yuv_to_rgb1 = input_conversion(input_layer1, nv12_to_rgb)    # or: i420_to_yuv1, yuv_to_rgb1 = input_conversion(input_layer1, i420_to_rgb)
Input Calibration frames compiler_optimization_level=0 compiler_optimization_level=max
RGB letterboxed OK, 3 contexts, 6.71 FPS on device OK, 2 h 01 min, 10.06 FPS on device
RGB native crops — OK, 2 h 01 min, 10.06 FPS on device
NV12 letterboxed OK, 3 contexts, 8.71 FPS on device Fail after 1 h 17 min
NV12 native crops OK, 17 min, 8.71 FPS on device Fail after 1 h 19 min
I420 native crops — Fail after 1 h 16 min

All three level-max failures are identical: valid multi-context partitions are found (Successful Multi Context Partition, ~12–13 min each), and the compile then aborts in validation:

yolov8s_det Successful Multi Context Partition (duration: 12m 54s)
#3 (1/28) Status: 43, Most common message: Validator failed on node: conv21 with Agent infeasible
Validator failed on node: conv21 with Agent infeasible
BackendAllocatorException: Compilation failed: Validator failed on node: conv21 with Agent infeasible

In the passing RGB level-max build the same message appears for a candidate (#3 (1/50) Status: 43, Most common message: Validator failed on node: concat2 with Agent infeasible), and the search continues and finishes. With YUV input it ends the compilation instead. Recalibrating doesn’t change it, and the result is the same for NV12 and I420. So it seems tied to the on-chip input conversion layers in the level-max flow.

What looks wrong to us

  1. Other limit violations during the search are treated as a rejected candidate, and the search continues: rows buffered (257) exceeds maximal (255), Automri finished with too many resources. Value doesn't fit in field instead aborts the whole compilation, although valid partitions (#102, #104) had already been found and could have been used.

  2. At =max a complete, valid partition is accepted and then can’t be encoded, so the check that rejects candidates doesn’t cover this limit.

  3. The message doesn’t say which field or layer overflows, so we can’t steer around it, for example with resources_param, buckets or per-context settings.

  4. With YUV input at level max, Agent infeasible on conv21 aborts the compile, while the same message on another node in an RGB build only rejects that candidate.

Questions

  1. Is Value doesn't fit in field meant to be fatal during partition search, or should the candidate be rejected like the row-buffer limit? Is there a flag to make it non-fatal, or to fall back to the best valid partition found so far?

  2. Which fields do 1951 and 119 refer to? Is there a debug log level that names the layer or context?

  3. Is there a supported way to make the compiler apply a specific partition from the search, e.g. the 4.63 FPS candidate, other than the .auto.alls of a successful compile?

  4. Is this fixed or changed in a DFC release after 5.3.0?

  5. What does Agent infeasible mean for conv21, and is on-chip nv12_to_rgb / i420_to_rgb input conversion supported together with compiler_optimization_level=max on Hailo-10H? Is there a setting that lets the level-max flow treat it as a rejected candidate, as it does for the RGB build?

Related threads: the same error text, on other models, mostly without an answer:

Thanks!

2

Hi thank you for posting the issues.
Can you attach the quantized har (the input of the compilation ) and also the full core logs of the run with level=max?
Should be easy to solve these issue in upcoming versions and also we can give you workarounds for making it compile on 5.3.0

3

Hi Omri, thanks, and sorry for the slow reply.

I’m preparing a repro HAR for you: the same YOLOv8s graph at 1×3×3040×4064 (2 classes, the same six head convolutions as end nodes, the same optimize script), with random weights and random calibration data, so I can share it openly. I’ll confirm it fails the same way (field 1951 at level 1, field 119 at level max, and the conv21 “Agent infeasible” with NV12 input at 3008×2240) and attach the quantized HAR together with the full hailo_sdk.core.log from the level-max runs.

In the meantime, any 5.3.0 workaround you can suggest would help: e.g. a way to fall back to the best valid partition, or settings that avoid the field overflow at this input size.

Separately, we found that the on-chip nv12_to_rgb on Hailo-10H scales the YUV→RGB gains by about 0.92–0.94, depending on input size, and that the DFC emulator doesn’t model it. I’ll post that as its own topic with a minimal repro and link it here, since it may be related to the YUV-input failure in part 2.

4

Hi Omri,

thanks for the reply. As you suggested, I’ve reproduced both failures with a model that can be shared:

Package: https://drive.google.com/file/d/1hD6qx3bKQ4SkbvFwI7gqLIp_QqNRb1vO/view?usp=sharing
hailo_dfc530_repro.tar.gz, 360 MB, sha256 starts with 2fe80e727dc5e675.

Model: YOLOv8s from Ultralytics’ public COCO yolov8s.pt, with the class head re-initialised for 2 classes (torch.manual_seed(0), no training, none of our weights). It’s exported to ONNX (opset 11, static input) and parsed with the six head convolutions as end nodes. Calibration is 64 crops of Ultralytics’ example images bus.jpg and zidane.jpg. Purely random weights failed earlier, in quantization, so I used the COCO backbone. The graph and optimize script are the same as in my original post: normalization /255, optimization_level 0, compression 0, calibset 64, a16_w16 on conv63.

Results with DFC 5.3.0, target hailo10h:

Run Input Compiler level Result
A 4064×3040 RGB 1 Abort during partition search: Value doesn't fit in field (1951)
A 4064×3040 RGB max Partition found, then fails: Value doesn't fit in field (119)
B 3008×2240, on-chip nv12_to_rgb max Fails: Agent infeasible

So all three failures from the original post show up with public weights too.

The package contains:

  • the two ONNX files with their manifests (end nodes);

  • per run, the quantized HAR (the compile input) and the optimize script;

  • per compile, the exact full.alls, compile.log, result.json and the DFC’s own logs (hailo_sdk.core.log etc.);

  • a README and SHA256SUMS.

To reproduce, load the *_opt.har and run load_model_script with the full.alls from the variant folder, then runner.compile().

The questions from the original post still stand. Most important for us: can Value doesn't fit in field be treated as a rejected candidate rather than a fatal error, and is NV12 input supported at compiler_optimization_level=max on Hailo-10H?

Thanks!