Hi Hailo team & community,
First, thanks for the hailocs/hailo-whisper toolkit — we used it to bring a custom Polish fine-tuned Whisper-base onto a Hailo-10H (RPi5) and it works great. We run the full pipeline on-device: encoder HEF + decoder HEF via HailoRT, with our own autoregressive loop (host-side token embedding + the 4-way split-logits matmul_split, exactly as in convert_whisper_decoder.py). End result: ~37.6% WER on our Polish eval set, fully on silicon.
We also applied one optimization: moving the final vocabulary projection off-chip (chip ends at /ln/Add_1 → host computes logits = hidden[i] @ token_embedding.T for the current position only). That shrank the decoder output ~100× and gave us a clean ~2× decoder speedup with no accuracy loss.
Our remaining bottleneck is decoder latency. The fixed-sequence decoder reprocesses the whole token window each step, so per-token cost stays high (measured ~80 ms/token for base, ~30 ms/token for tiny). The decoder dominates end-to-end latency.
We tried to build a single-step KV-cache decoder ourselves and export it for the DFC, but the parser rejects it — UnsupportedShuffleLayerError on the attention reshape as soon as we inject a cache tensor (we tried 4 variants: manual attention, SDPA, seq≥2, and pre-shaped 4D cache — all fail). It looks like the parser only accepts the standard fused-MHA pattern, which KV-cache inherently breaks.
Meanwhile, the GenAI Model Zoo Whisper-Base clearly runs with KV-cache (~43 ms/token, TPS 23.36), but the GenAI guide states “users can upload models: not supported” and those HEFs are Hailo-built.
Questions:
- Is there any supported path to compile our own fine-tuned Whisper weights into the GenAI HEF format (with KV-cache) used by Speech2Text / Hailo-Ollama?
- If not — is KV-cache / incremental decoding on the roadmap for the hailo-whisper custom-conversion path?
- Any recommended way to reduce per-token decoder latency for custom models beyond the matmul-on-host trick above?
Happy to share more details (compile configs, .alls, latency breakdown). Thanks!