This is a follow-up in the spirit of the earlier *“DFC conversion failures with transformer-based models (SwinV2, ViT, ConvNeXt)”* thread. The common takeaway from those reports — and from my own first attempt — was *“transformer graphs don’t survive the DFC optimizer.”* For **encoder** models I now believe that conclusion is partly wrong: the most cited crash (`IndexError` inside *LayerNorm Decomposition*) is **not** an optimizer bug. It is triggered by feeding **3D calibration data** to an input layer that the parsed HN holds in **4D**, and it is fixable in **one line**, with **no SDK patch**.
I’m posting the full root-cause trace, the fix, the quality numbers, and one genuine native-compiler wall I could **not** work around (`a16_w16` → `BackendAllocatorException`), in hopes it helps others and that Hailo engineers can confirm/extend it.
**Environment**
* Raspberry Pi 5 (aarch64, 16 GB) + Hailo-10H (8 GB LPDDR4), HailoRT **v5.3.0**
* Compilation host: x86_64, **DFC (`hailo_sdk_client`) v5.3.0** from `hailo_ai_sw_suite_2026-04:1`, TF 2.19.1, Python 3.10, **CPU only** (Blackwell / sm_120 GPUs are not usable by the Suite’s TF build — separate finding)
* Model: `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` (XLM-R, 12 layers, hidden 384, seq 256) — a sentence-embedding **encoder**
* * *
## 1. Summary of findings
| # | Finding | Status |
|—|—|—|
| 1 | The DFC parser elevates a 3D **NWC** transformer input to 4D **NHWC** (`H=1`) internally — the parsed HN body is 4D | **Undocumented** |
| 2 | The `IndexError: input_shape[3]` in *LayerNorm Decomposition* is caused by **3D calibration data**, not the optimizer | **Fixable, no patch** |
| 3 | One-line fix: feed calibration as 4D `[N,1,SEQ,HIDDEN]` (and mask as `[N,1,1,SEQ]`) | **Works** → HEF builds |
| 4 | The parser **rejects** a 4D `net_input_format` for the main input and breaks attention `Transpose` parsing — you must parse in 3D NWC | **Undocumented constraint** |
| 5 | Any `a16_w16` layer crashes the backend allocator with `BackendAllocatorException` — confirmed across 4 configs | **Native wall, not user-fixable** |
| 6 | HEF output sits in a **different channel basis** than ONNX → tensor-to-tensor cosine ~0.1 is misleading; use basis-invariant metrics | **Methodology note** |
* * *
## 2. The “wall”: `IndexError` in LayerNorm Decomposition
The crash everyone hits when running `optimize()` on a parsed encoder:
```
File “…/acceleras/…/slice_op.py”, line 143, in _validate_slices
... input_shape\[3\] ...
IndexError: list index out of range
```
The intuitive reading is *“the optimizer’s LayerNorm decomposition can’t handle this graph.”* It is more specific than that.
**Observation:** the parsed HN already holds the main input layer in **4D**. Instrumenting `base_atomic_op.compute_output_shape` shows:
```
input_layer1 in [[-1, 1, 256, 384]] out [[-1, 1, 256, 384]]
ew_add1 in [[-1, 1, 256, 384], [-1, 1, 256, 384]] out [[-1, 1, 256, 384]]
```
i.e. `[B, H=1, W=SEQ, C=HIDDEN]`. The parser **silently elevates** the 3D NWC sentence input to 4D NHWC with `H=1`. The entire decomposed LayerNorm body (35+ ops: `reduce`, `slice`, `conv_stripped`, `matmul`, `softmax`) is built for that 4D layout, with `reduce_axes=[3]` and `slice_op`/`conv_stripped_op` indexing `input_shape[3]` — the channel axis under NHWC.
**Root cause:** `stats_collector` rebuilds the Keras model following the **rank of the calibration data**, not the rank of the HN. The canonical compile scripts feed calibration as 3D `[N, SEQ, HIDDEN]`. That 3D build overrides the HN’s 4D, so the 4D-built ops receive 3D tensors where axis `3` does not exist → `IndexError`.
It is not the optimizer being unable to decompose the graph. It is a **shape-rank mismatch between the calibration data and the parsed HN**, surfacing inside the decomposition ops.
* * *
## 3. The fix (one line)
Feed calibration in **4D**, matching the HN:
```python
# emb = word+position+token_type embedding sum, shape [N, SEQ, HIDDEN]
# msk = raw 0/1 attention mask, shape [N, SEQ]
calib = {
f"{MODEL_NAME}/input_layer1": emb\[:, None, :, :\], # \[N, 1, SEQ, HIDDEN\] <-- the fix (was \[N, SEQ, HIDDEN\])
f"{MODEL_NAME}/input_layer2": msk\[:, None, None, :\], # \[N, 1, 1, SEQ\]
}
runner.optimize(calib) # passes LayerNorm Decomposition, Statistics Collector, etc.
hef = runner.compile() # 8-bit: builds, 13 contexts, ~19.4 MB
```
With this, on a **stock SDK** (no source modifications), `optimize()` and `compile()` both run to completion and produce a valid HEF. The change is purely in the calibration tensors handed to `optimize()`.
* * *
## 4. Undocumented parser constraint: parse in 3D NWC, **not** 4D
A natural reaction to finding §2 is *“then declare the input as 4D at parse time.”* The parser rejects this:
```
Input format [batch, height, width, channels] doesn’t match its rank (3)
```
and, if forced, it also breaks the attention `Transpose` parsing:
```
UnsupportedShuffleLayerError
```
The correct parse for an encoder is **3D NWC** (matching the canonical `bert_base_uncased` in the Model Zoo):
```python
start_node_names = [“/embeddings/Add_1”, “/Mul”]
net_input_shapes = {“/embeddings/Add_1”: [B, SEQ, HIDDEN], “/Mul”: [B, 1, 1, SEQ]}
net_input_format = {“/embeddings/Add_1”: “NWC”, “/Mul”: “NHWC”}
```
The parser does the 3D→4D elevation itself. **That elevation is the part that’s undocumented and is the source of the confusion** — the HN you parse is 4D even though you handed it a 3D format.
**Note on batch:** the HN batch is dynamic (`-1`). The `batch=16` that matters is the **calibration** batch (`model_optimization_config(calibration, batch_size=16, …)`), which is not baked into the HEF. You do **not** need to re-export the ONNX at static batch>1.
* * *
## 5. The real wall I could not pass: `a16_w16` → `BackendAllocatorException`
8-bit compiles cleanly, but its embedding quality is degraded (see §6). The obvious fix is 16-bit on the quantization-sensitive attention `softmax`. Two problems appear, and the second is a hard native wall.
**5.1 Quantization.** Without a 16-bit softmax, `model_optimization_config(negative_exponent, rank=0)` fails:
```
NegativeSlopeExponentNonFixable: … ew_sub_softmax …
Desired shift is 8.0, but op has only 8 data bits
```
This is fixed by putting the softmax at `a16_w16`.
**5.2 Backend allocator (the wall).** With **any** `a16_w16` layer present, `compile()` crashes — *after* finding valid partitions:
```
[info] Found valid partition to 4 contexts, Performance improved by 2.7%
[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed with unexpected crash
```
I tried hard to rule out a configuration cause. **Four independent compiles, same crash:**
| # | 16-bit configuration | Result |
|—|—|—|
| 1 | `a16_w16` on softmax + ew_add | `BackendAllocatorException` |
| 2 | `a16_w16` on softmax only | `BackendAllocatorException` |
| 3 | softmax + `model_optimization_flavor(compression_level=4)` (81% of weights → 4-bit) | `BackendAllocatorException` |
| 4 | softmax + compression 4 + `performance_param(compiler_optimization_level=max)` | `BackendAllocatorException` |
This rules out the three reasonable config hypotheses:
* **Not memory footprint.** `compression_level=4` (which the compiler itself recommends for >10M-param models; this one is 21.41M) pushes 81% of weights to 4-bit and still crashes.
* **Not allocator search depth.** `compiler_optimization_level=max` forces the exhaustive search — it found tighter 2–3-context partitions — and still crashes.
* **Not a specific layer.** softmax-only and softmax+ew_add behave identically.
The trigger is the **mere presence of an `a16_w16` layer in this graph**. Unlike the §2 crash (a calibration-rank issue, fixed in user code), this one lives in the closed native graph compiler:
```
hailo_sdk_client/allocator/hailo_tools_runner.py → graph compiler binary
```
so there is no user-side workaround. **For encoders on DFC v5.3.0, 8-bit is the ceiling.** This is the one place I’d ask Hailo to look.
* * *
## 6. Quality methodology: do **not** compare HEF output tensor-to-tensor
A trap worth flagging. Comparing the HEF `last_hidden_state` against the ONNX FP32 output gives cosine **~0.1**, which looks like a broken conversion. It is not. `SDK_NATIVE` (full precision emulation) **also** gives ~0.1 against ONNX — the HEF output lives in a **different channel basis** (Hailo reorders channels internally). The basis difference cancels in any basis-invariant operation.
**Use basis-invariant metrics only:** top-k retrieval, similarity-matrix correlation.
On a controlled ES/EN translation-pair retrieval test (12 pairs = 24 sentences, each sentence’s correct neighbor is its translation):
| Model | top-1 retrieval | sim-matrix corr. vs FP32 |
|—|—|—|
| FP32 (reference) | 100% (24/24) | 1.00 |
| **HEF 8-bit (BIT_EXACT)** | **70.8% (17/24)** | 0.37 |
The 8-bit HEF works (70.8% ≫ ~4% random over 23 candidates) but degrades. On a real, **homogeneous** technical document corpus the degradation is worse — fine-grained discrimination collapses (anisotropy) and retrieval becomes unusable, while the 6.5× throughput advantage is real. Net: for quality-sensitive multilingual retrieval, CPU FP32 remained the better choice **for me**; the 8-bit HEF is viable where you need to offload the CPU and can tolerate the quality drop. A 16-bit HEF would likely close the gap — hence §5 matters.
* * *
## 7. Reproduction
Core of the working compile (clean SDK, no patches):
```python
from hailo_sdk_client import ClientRunner
runner = ClientRunner(hw_arch=“hailo10h”, har=“multilingual_parsed.har”) # input_layer1 already 4D
runner.load_model_script(“”"
model_optimization_config(calibration, batch_size=16, calibset_size=1024)
pre_quantization_optimization(equalization, policy=enabled)
model_optimization_flavor(optimization_level=0, compression_level=0)
set_input_mask_to_softmax()
“”")
emb, msk = gen_calib(1024) # real multilingual embedding sums, CPU
calib = {
"multilingual_minilm_l12_v2/input_layer1": emb\[:, None, :, :\], # 4D <-- THE FIX
"multilingual_minilm_l12_v2/input_layer2": msk\[:, None, None, :\],
}
runner.optimize(calib)
open(“multilingual_minilm.hef”, “wb”).write(runner.compile())
```
Inference input contract: `input_layer1 = word+pos+type embedding sum` (4D `[N,1,SEQ,HIDDEN]`), `input_layer2 = raw 0/1 attention mask` (4D `[N,1,1,SEQ]`). Output `last_hidden_state` → squeeze `H` → masked mean-pool → L2-normalize. Build the vector index **entirely** from HEF embeddings (do not mix HEF and FP32 vectors, because of the basis difference in §6).
* * *
## 8. Questions / requests for Hailo
1. Is the 3D→4D (`H=1`) elevation of NWC transformer inputs intended, and is the requirement that **calibration data match the 4D HN rank** documented anywhere? It is the single thing that unblocks encoder compilation.
2. Is the `BackendAllocatorException` on **any** `a16_w16` layer a known limitation of the v5.3.0 graph compiler for multi-context transformer graphs? Is there a config that lets 16-bit attention through, or is it fixed in a later DFC?
3. Could the parser emit a clearer error than `IndexError: input_shape[3]` when calibration rank ≠ HN rank? It sends everyone toward “transformers unsupported.”
**Environment:** RPi5 (aarch64) + Hailo-10H, HailoRT 5.3.0; compile host x86_64, DFC v5.3.0 (`hailo_ai_sw_suite_2026-04:1`), TF 2.19.1, CPU-only. Model: `paraphrase-multilingual-MiniLM-L12-v2` (XLM-R, 12 layers).