We hit what appears to be the same issue, and we managed to isolate it to a **minimal, shareable reproduction**. Posting our findings here in case it helps.
## Environment
- Hailo AI SW Suite 2025-10 (DFC 3.33.0), HailoRT 4.23.0
- Device: Hailo-8 (Raspberry Pi 5 + AI HAT), driver/firmware matching 4.23.0
- Compilation on x86_64 Ubuntu 20.04 (Docker suite container)
## Symptom (production model)
SSD-style detector with a MobileNetV3 backbone and an FPN-like top-down path using three chained `ConvTranspose` layers (kernel 2x2, stride 2, pad 0; e.g. 512→960 ch). The model parses, quantizes and compiles without errors. On device, however:
- Output branches fed by the deconvs: correlation **0.05–0.37** against `SDK_QUANTIZED` emulation of the compiled HAR (same input)
- Output branches not touching any deconv: correlation **0.97–0.999**
- The corrupted tensors contain many values pinned at the quantization range endpoints (uint codes 0x00 / max), i.e. padding-like garbage mixed into real data
We verified this is not an application issue (identical corrupted values via both the C++ `InferModel` API and pyhailort `InferVStreams`) and not allocation-specific (identical failure with the default compilation, `resources_param` variations, and a 5-hour `performance_param(compiler_optimization_level=max)` build).
We also see `Microcode violation: DUAL_BUFFER_FLIP_OUT in the next N cycles after APU_WRITE_TO_MEM` in `deconv*_defuse_*` layers during compilation (info level), like the OP. Note however that the same violations appear for layers that work correctly (e.g. `resize`), so they don’t seem diagnostic by themselves.
## Minimal reproduction (scripts inlined in the appendix below; random weights, no proprietary data)
`make_repro_onnx_v2.py` builds a small random-weight ONNX: conv backbone → three chained ConvTranspose (512→960, 960→112, 112→40, all k2/s2/pad0) with skip-adds, one head fed by the deconv chain (`head_deconv`) and one control head fed by convs only (`head_ctrl`). Both heads output 40x40x54.
```
python3 make_repro_onnx_v2.py
hailo parser onnx deconv_repro_v2.onnx --hw-arch hailo8 --har-path deconv_repro_v2.har -y
hailo optimize deconv_repro_v2.har --calib-set-path calib_random.npy --model-script repro.alls --hw-arch hailo8 --output-har-path deconv_repro_v2_q.har
# single-context build:
hailo compiler deconv_repro_v2_q.har --hw-arch hailo8
# multi-context build (7 contexts, forced via resources_param):
hailo compiler deconv_repro_v2_q.har --hw-arch hailo8 --model-script compile_mc.alls --output-dir v2mc
```
Run each HEF on device with a fixed input (`pi_repro_test.py`) and compare against `SDK_QUANTIZED` emulation (`repro_crosscheck_v2.py`):
| Build | head_ctrl (no deconv) | head_deconv |
|—|—|—|
| Single context | corr +1.000 | corr **+1.000** (OK) |
| Multi context (7) | corr +1.000 | corr **+0.585** (corrupted) |
So the trigger seems to be **deconv output/intermediate tensors crossing context boundaries**: the same quantized network is bit-exact vs emulation in a single-context build and corrupted in a multi-context build. Our production model always compiles to 4–6 contexts, which explains why it is always corrupted regardless of partitioning.
## Workaround that fully fixed it for us
Since ConvTranspose with kernel==stride (no overlap) is mathematically identical to a pointwise conv + pixel shuffle, we rewrote the ONNX:
- `ConvTranspose(C_in→C_out, k2, s2, pad0)` → `Conv 1x1 (C_in→4*C_out)` + `DepthToSpace(blocksize=2, mode=DCR)`
- Weight rearrangement only (no retraining): `W1[(dy*2+dx)*C_out + co, ci] = W[ci, co, dy, dx]`
- Rewritten ONNX matches the original to 3e-7 relative error (onnxruntime)
The DFC parses DepthToSpace into its native `depth_to_space` layer, and the compiled multi-context model now matches emulation on device (corr 0.97–0.999 on all outputs).
Questions: is this a known issue, and is it fixed in a newer DFC release? The full repro package is inlined below (the forum does not allow script attachments). Happy to run additional experiments.
## Appendix: reproduction scripts
### make_repro_onnx_v2.py
```python
# Repro model v2: three chained deconvs with skip-adds (mirrors the production model),
# extra heavy convs to allow forcing a multi-context compilation
# Outputs: head_deconv (fed by the deconv chain, 40x40x54) / head_ctrl (conv-only control, 40x40x54)
import numpy as np
import onnx
from onnx import helper, numpy_helper, TensorProto
rng = np.random.RandomState(7)
nodes, inits = [], []
def conv(name, inp, cin, cout, k, s, relu=True):
w = (rng.randn(cout, cin, k, k) \* np.sqrt(2.0 / (cin \* k \* k))).astype(np.float32)
b = np.zeros(cout, dtype=np.float32)
inits.append(numpy_helper.from_array(w, f'{name}\_w'))
inits.append(numpy_helper.from_array(b, f'{name}\_b'))
pad = k // 2
nodes.append(helper.make_node('Conv', \[inp, f'{name}\_w', f'{name}\_b'\], \[f'{name}\_y'\],
name=name, kernel_shape=\[k, k\], strides=\[s, s\],
pads=\[pad, pad, pad, pad\]))
if relu:
nodes.append(helper.make_node('Relu', \[f'{name}\_y'\], \[f'{name}\_r'\], name=f'{name}\_relu'))
return f'{name}\_r'
return f'{name}\_y'
def deconv(name, inp, cin, cout):
w = (rng.randn(cin, cout, 2, 2) \* 0.05).astype(np.float32)
inits.append(numpy_helper.from_array(w, f'{name}\_w'))
nodes.append(helper.make_node('ConvTranspose', \[inp, f'{name}\_w'\], \[f'{name}\_y'\],
name=name, kernel_shape=\[2, 2\], strides=\[2, 2\],
pads=\[0, 0, 0, 0\], dilations=\[1, 1\]))
nodes.append(helper.make_node('Relu', \[f'{name}\_y'\], \[f'{name}\_r'\], name=f'{name}\_relu'))
return f'{name}\_r'
def add(name, a, b):
nodes.append(helper.make_node('Add', \[a, b\], \[f'{name}\_y'\], name=name))
return f'{name}\_y'
x = ‘input’
x = conv(‘c1’, x, 3, 32, 3, 2) # 160
x = conv(‘c2’, x, 32, 64, 3, 2) # 80
f40 = conv(‘c3’, x, 64, 128, 3, 2) # 40x40x128
f20 = conv(‘c4’, f40, 128, 256, 3, 2) # 20x20x256
f10 = conv(‘c5’, f20, 256, 512, 3, 2) # 10x10x512
# weight filler (~7M params) so resource caps can force a multi-context split
f10 = conv(‘h1’, f10, 512, 512, 3, 1)
f10 = conv(‘h2’, f10, 512, 512, 3, 1)
f10 = conv(‘h3’, f10, 512, 512, 3, 1)
f5 = conv(‘c6’, f10, 512, 512, 3, 2) # 5x5x512
f5 = conv(‘h4’, f5, 512, 512, 3, 1)
f5 = conv(‘h5’, f5, 512, 512, 3, 1)
# chained deconv top-down path (same channel configuration as the production model)
d1 = deconv(‘deconv1’, f5, 512, 960) # 10x10x960
p1 = conv(‘proj1’, f10, 512, 960, 1, 1, relu=False)
m1 = add(‘add1’, d1, p1)
d2 = deconv(‘deconv2’, m1, 960, 112) # 20x20x112
p2 = conv(‘proj2’, f20, 256, 112, 1, 1, relu=False)
m2 = add(‘add2’, d2, p2)
d3 = deconv(‘deconv3’, m2, 112, 40) # 40x40x40
p3 = conv(‘proj3’, f40, 128, 40, 1, 1, relu=False)
m3 = add(‘add3’, d3, p3)
outA = conv(‘head_deconv’, m3, 40, 54, 3, 1, relu=False) # 40x40x54 (deconv-fed)
outB = conv(‘head_ctrl’, f40, 128, 54, 3, 1, relu=False) # 40x40x54 (control)
graph = helper.make_graph(
nodes, 'deconv_repro_v2',
\[helper.make_tensor_value_info('input', TensorProto.FLOAT, \[1, 3, 320, 320\])\],
\[helper.make_tensor_value_info(outA, TensorProto.FLOAT, \[1, 54, 40, 40\]),
helper.make_tensor_value_info(outB, TensorProto.FLOAT, \[1, 54, 40, 40\])\],
inits)
model = helper.make_model(graph, opset_imports=[helper.make_opsetid(‘’, 13)])
model.ir_version = 7
onnx.checker.check_model(model)
onnx.save(model, ‘/local/shared_with_docker/repro_deconv/deconv_repro_v2.onnx’)
nparams = sum(np.prod(i.dims) for i in inits)
print(f’saved deconv_repro_v2.onnx params={nparams/1e6:.1f}M’)
```
### repro.alls
```text
model_optimization_flavor(optimization_level=0, compression_level=0)
```
### compile_mc.alls
```text
resources_param(max_control_utilization=0.15, max_compute_utilization=0.15, max_memory_utilization=0.15)
```
### pi_repro_test.py
```python
# Run the repro HEF on the device and save all outputs
# Usage on device: python3 pi_repro_test.py [hef_path]
import numpy as np
from hailo_platform import (HEF, VDevice, InferVStreams, ConfigureParams,
InputVStreamParams, OutputVStreamParams,
FormatType, HailoStreamInterface)
import sys
hef_path = sys.argv[1] if len(sys.argv) > 1 else ‘deconv_repro.hef’
hef = HEF(hef_path)
rng = np.random.RandomState(123)
rgb = rng.randint(0, 256, (1, 320, 320, 3), dtype=np.uint8) # deterministic input (fixed seed)
np.save(‘repro_input.npy’, rgb)
with VDevice() as target:
cfg = ConfigureParams.create_from_hef(hef=hef, interface=HailoStreamInterface.PCIe)
ng = target.configure(hef, cfg)\[0\]
in_params = InputVStreamParams.make(ng, format_type=FormatType.UINT8)
out_params = OutputVStreamParams.make(ng, format_type=FormatType.FLOAT32)
in_name = hef.get_input_vstream_infos()\[0\].name
def run():
with InferVStreams(ng, in_params, out_params) as pipe:
return pipe.infer({in_name: rgb})
try:
with ng.activate():
res = run()
except Exception:
res = run()
np.savez(‘pi_repro_outputs.npz’, **{k: np.asarray(v) for k, v in res.items()})
print(‘saved pi_repro_outputs.npz’)
for k, v in res.items():
print(k, np.asarray(v).shape)
```
### repro_crosscheck_v2.py
```python
# Compare device outputs (pi_repro_outputs.npz) against SDK_QUANTIZED emulation.
# Heads are labeled deconv-fed / control by graph reachability from deconv layers.
import numpy as np
from hailo_sdk_client import ClientRunner, InferenceContext
DIR = ‘/local/shared_with_docker/repro_deconv’
HAR = f’{DIR}/v2mc/deconv_repro_v2_compiled.har’
rgb = np.load(f’{DIR}/repro_input.npy’).astype(np.float32)
runner = ClientRunner(har=HAR)
hn = runner.get_hn_dict()
succ = {}
for name, layer in hn[‘layers’].items():
for i in (layer.get('input') or \[\]):
succ.setdefault(i, \[\]).append(name)
seen, stack = set(), [n for n, l in hn[‘layers’].items() if l[‘type’] == ‘deconv’]
while stack:
n = stack.pop()
for s in succ.get(n, \[\]):
if s not in seen:
seen.add(s); stack.append(s)
deconv_reach = seen
out_src = {}
for n, l in hn[‘layers’].items():
if l\['type'\] == 'output_layer':
out_src\[l\['input'\]\[0\]\] = ('deconv-fed' if n in deconv_reach or l\['input'\]\[0\] in deconv_reach else 'control')
with runner.infer_context(InferenceContext.SDK_QUANTIZED) as ctx:
outs = runner.infer(ctx, rgb)
emu_list = [np.asarray(v).reshape(-1) for v in (outs if isinstance(outs, list) else [outs])]
pi = np.load(f’{DIR}/pi_repro_outputs.npz’)
def corr(a, b):
return float(np.corrcoef(a, b)\[0, 1\])
# print correlation of each device output against every emulator output
for k in pi.files:
dev = pi\[k\].reshape(-1)
cs = \[corr(e, dev) for e in emu_list\]
conv = k # vstream name == conv layer name
kind = out_src.get(conv, '?')
print(f'{k} \[{kind}\]: corr vs emulator outputs = {\[f"{c:+.3f}" for c in cs\]} best={max(cs):+.4f}')
```