Hailo-10H nv12_to_rgb on-chip conversion runs ~6–8 % low in gain vs DFC emulation; the correction depends on input size and on the model

Hi all,

we run YOLOv8 detection and pose models on a Hailo-10H with on-chip NV12 input conversion (input_conversion(input_layer1, nv12_to_rgb) in the model script). On the chip, the converted image comes out darker and lower in contrast than the DFC emulates: every gain of the BT.601 matrix is about 6–8 % low, while the offsets are right. The DFC’s emulation uses the correct matrix, so its accuracy and SNR estimates never show it; the loss appears only on hardware.

We work around it on the host, but the right correction is different for each input size and, on real models, for each HEF. We’d like to know whether this is expected, and whether there’s a supported way to make the chip and the emulation agree.

Environment

  • DFC 5.3.0 (hailo_sdk_client 5.3.0, Docker image with CUDA 12.5.1), target hailo10h.

  • HailoRT 5.3.0 on a Raspberry Pi 5 (Hailo-10H), InferModel API, UINT8 input.

  • Model scripts: normalization([0,0,0],[255,255,255]), input_conversion(..., nv12_to_rgb), calibration set in BT.601 limited-range YUV 4:4:4.

1. Probe: the conversion itself, per input size

A probe HEF per size: NV12 input → nv12_to_rgb → /255 → identity 1×1 conv, so the output is the chip’s RGB. Fed random YUV, flat patches and real frames; least-squares fit per output channel and 2×2 pixel phase.

Input size Chip gain Y / V→R / U→B (BT.601 = 1) Host correction (1 / mean)
640×640 0.918 / 0.927 / 0.923 ×1.085
1280×736 0.938 / 0.947 / 0.942 ×1.061
2048×1536 0.930 / 0.930 / 0.934 ×1.074
4064×3040 0.935 / 0.934 / 0.938 ×1.068

Otherwise the conversion is exact: linear (fit rms 0.30, i.e. 8-bit rounding), offsets as BT.601 (R −223.1, G +135.9, B −276.8), chroma nearest with correct siting, identical across the frame width. The gain does not follow the input size smoothly (1280×736 needs less correction than both 640 and 2048).

2. Real models: the best correction is per HEF

Correction: every NV12 byte through a host LUT v → min(255, round(v × g)) before inference. For each HEF we sweep g and compare the chip’s raw output heads with the same model’s ONNX on the same frames (mean SNR over box and keypoint heads):

Model Input Best g Probe g for that size
YOLOv8s-pose, model A 1280×736 1.0851 1.061
YOLOv8s-pose, model B 1280×736 1.0876 1.061
YOLOv8s-pose, model C (two calibration sets) 1280×736 1.0851 1.061
YOLOv8n detection 1280×736 1.061 1.061
YOLOv8n-pose, model D 2048×1536 1.074 1.074
YOLOv8n-pose, model E 2048×1536 1.06 (flat to 1.07) 1.074
YOLOv8n-pose 4064×3040 1.068 1.068
  • At the same input size, different models need different corrections (1.061 vs 1.085 at 1280×736; 1.06 vs 1.074 at 2048×1536).

  • The optimum is narrow: 0.5 % off costs 3–8 dB on the output heads. Without any correction a 4064×3040 nano model matches 38 of 69 of its ONNX detections; with the right g, 68 of 69.

  • An RGB-input HEF of the same model, fed the same frames converted to RGB on the host, matches its ONNX without any correction, so 4:2:0 chroma itself costs nothing.

Our guess is that the DFC folds the input conversion into the first quantized layer, so the hardware’s fixed conversion error lands differently in each model’s quantization.

3. Folding the inverse into the model script doesn’t help

We built a HEF with the inverse gains folded into normalization (and the calibration set scaled to match). On the chip it behaved exactly like the uncorrected build: the hardware conversion seems to use its own fixed coefficients regardless of the normalization folded around it.

Questions

  1. Is the gain difference between the on-chip nv12_to_rgb and the DFC emulation known? Is it fixed in a later DFC/HailoRT release?

  2. Which coefficients does the hardware conversion use, and why do they differ by input size?

  3. Is there a supported way to make chip and emulation agree, e.g. a model-script option, a different conversion layer, or calibrating with the chip’s coefficients?

  4. If a host-side correction is the intended way, is there a recommended method to find it per HEF?

We can share the probe HEFs and the probe script (identity network, no proprietary weights) if that helps.

Thanks!

Hi Simon,

Thank you for such a thorough write-up. The probe HEFs and the per-size and per-model tables make this much easier to investigate. We’d like to take you up on your offer: please share the probe HEFs and the probe script, either here or by direct message. We will pass them to our compiler team to confirm what coefficients the hardware conversion actually applies and why they change with input size.

While that is being investigated, a few points that may help.

Whether this is expected. No. The intent is that the on-chip conversion matches the DFC emulation. Your measurements show the offsets match the documented BT.601 limited-range values exactly, so the chip is using the right formula, but the gains come out low. That points to how the conversion is quantized and folded into the first layers of each model rather than a wrong matrix. It is also consistent with your finding that the correction differs per HEF. We’ll confirm this with R&D.

A note on DFC 5.4.0. In 5.4.0, the default yuv_to_rgb conversion, and the YUV stage of nv12_to_rgb, moved to Rec.601 full-range coefficients to align with the Hailo 15 ISP’s NV12 output. That change does not address the gain gap you’re seeing. However, since your pipeline uses limited-range BT.601, if you upgrade you should build the conversion explicitly as nv12_to_hailo_yuv followed by yuv601_to_rgb. That keeps your current limited-range behavior instead of switching silently to full range.

Calibration format. The guide notes that when a model includes the nv12_to_rgb hybrid conversion, an RGB calibration set can be used and the conversion is handled automatically during optimization. Could you try one of your 1280×736 pose models with an RGB calibration set instead of YUV 4:4:4 and see whether the chip/ONNX gap changes? If the conversion layer’s statistics were being derived from data in an unexpected format, that could explain why each model ends up with a different effective gain.

Comparing chip and emulation directly. To isolate the hardware conversion from ordinary quantization error, you can run the same YUV frames through runner.infer_context(InferenceContext.SDK_QUANTIZED) and InferenceContext.SDK_HAILO_HW with the compiled HAR. Any difference between those two outputs is exactly the chip-versus-emulation gap. It is also a more reliable way to tune your per-HEF host correction than sweeping against ONNX.

Your host-side LUT. Your workaround is reasonable as an interim measure. We’d rather find the root cause than have you maintain a per-HEF correction table, so we’ll follow up here once we have an answer on the coefficients and on whether a fix is planned.

Thanks again for the detailed report.

Hi Victor,

Thanks, that’s very helpful. The forum only accepts images, so the probe pack is here: [https://drive.google.com/file/d/1KlVOFKjOOiFL4j6fWe_bBam4w8Q8NFUo/view?usp=sharing\] (hailo_nv12_probe_20261106.tar.gz, 48 KB, sha256 e3c73ac87410…). It has the five probe HEFs (640×640 NV12 and I420, 1280×736, 2048×1536, 4064×3040), their exact model scripts, the identity ONNX files, the probe script with its options, and a README with the method and the numbers above. The probes were calibrated on our own frames, which aren’t included; the random-YUV and flat-patch tests in the script need no data.

RGB calibration: a rebuild is running now on one of our 1280×736 models (YOLOv8s detection, one class). It’s identical to its existing build (same ONNX, optimization level 2, nv12_to_rgb, one 16-bit output layer, compiler level max), except that the calibration is the RGB array instead of the BT.601 YUV 4:4:4 one, with the same 512 frames. The existing build’s best host gain is ×1.0851, against ×1.061 from the 1280×736 probe. I’ll post the new build’s gain sweep and head SNRs here.

SDK_QUANTIZED vs SDK_HAILO_HW: our DFC host has no Hailo device attached, and the Hailo-10H is on a Raspberry Pi 5 with HailoRT only. Instead, I’ll run SDK_QUANTIZED on the DFC host and HailoRT on the Pi over the same NV12 frames, and compare the raw outputs. That gives the chip-vs-emulation gap directly, without ONNX in between.

DFC 5.4.0: noted. We’ll build the conversion explicitly as nv12_to_hailo_yuv + yuv601_to_rgb when we move, to stay on limited range.

Thanks again!

Hi Simon,

A couple of things that will help both of your experiments give a clean answer:

SDK_QUANTIZED vs HailoRT comparison. Your plan works well. Two details are worth checking so the comparison is like for like:

  • Input. With nv12_to_rgb, the NV12→YUV format stage isn’t part of emulation by default, so SDK_QUANTIZED expects YUV 4:4:4 rather than NV12. Please convert the same NV12 frames to 4:4:4 on the host, with nearest-neighbor chroma, the same way the chip does it, before feeding the emulator.
  • Output. SDK_QUANTIZED returns dequantized values, while HailoRT returns raw quantized values unless asked otherwise. Setting the outputs to FLOAT32 in your InferModel configuration, or dequantizing with each output’s quantization info, will put both sides on the same scale.

The I420 probe. It would be very useful to see the gain fit for your 640×640 I420 probe next to the NV12 one, if you have it. Both conversions share the same YUV-to-RGB stage, so this tells us quickly where the gain is lost:

  • If I420 shows the same ~0.92 gain, the issue is in the color-conversion stage.
  • If I420 doesn’t, the issue is specific to the NV12 path.

Thanks Victor, calibrating with RGB data was the answer.

We rebuilt the 1280x736 identity probe twice from the same 64 calibration frames, once with RGB and once with the YUV (BT.601) version of those frames, and changed nothing else (level 0, compiler optimization max, nv12_to_rgb).

Calibration Y gain V→R U→B Relative to BT.601
RGB 1.156–1.167 1.601 2.018 0.99–1.00
YUV 1.069 1.479 1.860 0.92

With RGB calibration the on-chip conversion matches BT.601 limited range, so no host-side gain is needed. With YUV calibration the gain also depends on which frames are used: our earlier probe, calibrated with a different YUV set, came out at 0.94. The RGB-calibrated probe’s output range is wider (quantization scale 0.0125 vs 0.0040), so its fit residual is higher (rms 1.3 vs 0.3 in 0–255 units).

We saw the same on a real detection network: the RGB-calibrated build needs no input gain (best ×0.998, against ×1.085 for the YUV-calibrated build), and its confidences sit closer to the float ONNX model.

We also compared i420_to_rgb with nv12_to_rgb on the 640x640 probes, given the same image: the chip outputs are bit-identical.

Following your suggestion, we compared SDK_QUANTIZED with the chip on the same frames, for both 1280x736 identity probes from my last post (same 64 calibration frames, RGB vs YUV calibration). The emulator is fed full-size YUV 4:4:4 at the yuv_to_rgb layer, with the chroma repeated 2x2 from the NV12 bytes the chip received.

Probe Y gain, SDK_FP_OPTIMIZED Y gain, SDK_QUANTIZED Y gain, chip
YUV calibration 1.164 1.161 1.069
RGB calibration 1.164 1.158–1.167 1.156–1.167

(BT.601 limited: 1.164; V→R and U→B scale the same way.)

With RGB calibration, the chip matches SDK_QUANTIZED to 0.03 in 0–255 units, with identical fit residuals. With YUV calibration, SDK_QUANTIZED still gives BT.601 (0.997), but the chip runs at 0.918 in all three channels. So the emulator does not predict the chip in the YUV-calibrated case. The gain drop seems to come in after optimization, when the quantized conversion layer is compiled for the hardware. We can share the optimized HARs, the HEFs and the emulator script for these two probes if your team wants to reproduce it.