Exploring real-time neural inference on Hailo-8 for a multilingual cultural radio project

[giladn]

@Dietrich_Tollner Welcome to Hailo’s community!
Checkout this project by @Katrin_Tomanek

Real-time ASR on Raspberry Pi + Hailo8L with Whisper

Hey everyone, I’ve built on top of Hailo’s release of their Whisper model conversion and found that running a hybrid inference mode with the encoder running on the Hailo 8L and the decoder on CPU (in my case a Raspberry Pi 5) gives actually great real-time ASR performance and allows for live captioning (with ~250ms refresh time). The hybrid mode works much better than relying only on the Hailo 8L for whisper inference (it is really slow in autoregessive decoding without KV cache). I’ve releas…