KV-cache / low-latency decoding for custom fine-tuned Whisper on Hailo-10H?

General Discussion
1

Hi Hailo team & community,

First, thanks for the hailocs/hailo-whisper toolkit — we used it to bring a custom Polish fine-tuned Whisper-base onto a Hailo-10H (RPi5) and it works great. We run the full pipeline on-device: encoder HEF + decoder HEF via HailoRT, with our own autoregressive loop (host-side token embedding + the 4-way split-logits matmul_split, exactly as in convert_whisper_decoder.py). End result: ~37.6% WER on our Polish eval set, fully on silicon.

We also applied one optimization: moving the final vocabulary projection off-chip (chip ends at /ln/Add_1 → host computes logits = hidden[i] @ token_embedding.T for the current position only). That shrank the decoder output ~100× and gave us a clean ~2× decoder speedup with no accuracy loss.

Our remaining bottleneck is decoder latency. The fixed-sequence decoder reprocesses the whole token window each step, so per-token cost stays high (measured ~80 ms/token for base, ~30 ms/token for tiny). The decoder dominates end-to-end latency.

We tried to build a single-step KV-cache decoder ourselves and export it for the DFC, but the parser rejects it — UnsupportedShuffleLayerError on the attention reshape as soon as we inject a cache tensor (we tried 4 variants: manual attention, SDPA, seq≥2, and pre-shaped 4D cache — all fail). It looks like the parser only accepts the standard fused-MHA pattern, which KV-cache inherently breaks.

Meanwhile, the GenAI Model Zoo Whisper-Base clearly runs with KV-cache (~43 ms/token, TPS 23.36), but the GenAI guide states “users can upload models: not supported” and those HEFs are Hailo-built.

Questions:

  1. Is there any supported path to compile our own fine-tuned Whisper weights into the GenAI HEF format (with KV-cache) used by Speech2Text / Hailo-Ollama?
  2. If not — is KV-cache / incremental decoding on the roadmap for the hailo-whisper custom-conversion path?
  3. Any recommended way to reduce per-token decoder latency for custom models beyond the matmul-on-host trick above?

Happy to share more details (compile configs, .alls, latency breakdown). Thanks!

2

Thanks for this — it is the only public report I can find of a custom fine-tune running on a 10H, and we are trying to do the same thing and cannot get past the parser.

We have a Whisper-Small fine-tuned on custom audio segments, exported with the samehailo-whisper toolkit (patched fork, opset 13, 10 s window, 48-token decode).
ClientRunner(hw_arch="hailo10h").translate_onnx_model() fails on the encoder, for both small and tiny, identically on DFC 5.1.0, 5.2.0, 5.3.0 and 5.4.0:

Invalid kernel shape for base conv layer base_conv3 (translated from MatMul_2).
Kernel features: 768  Input features: 500  Groups: 0

The sequence length is being read as a channel count.

Three questions, if you have the time:

  1. Which DFC version built your base encoder and decoder HEFs? Hailo support has said whisper conversion “needs at least DFC 3.31.0” because of parser changes
    ( HEF files for Whisper-base compatible with HailoRT 4.19.0 (Hailo-8, amd64) ),
    so the version clearly matters, and 5.x may not carry them for hailo10h.
  2. Did you run the stock conversion/convert_whisper_encoder.py, or did you have to change anything — model script, translate_onnx_model arguments, input-shape declaration?
  3. Have you tried Whisper-Small? Base and small differ in d_model (768 vs 512) and layer count, and ours fails at the first attention MatMul rather than at conv1.

On your KV-cache question: we hit the same UnsupportedShuffleLayerError on attention reshapes for a different reason and reached the same conclusion, the parser only likes the fused MHA pattern.

Happy to share our ONNX, calibration set and exact commands if useful.