Hi, thanks for notifications.
Are there any plans to support Hailo conversion for models larger than Base (e.g., Small, Medium, Large)?
@hyungjun_Byun support to larger Whisper models is planned for Hailo-10H architecture, which allows faster throughput when running those models.
For Hailo-8, larger models may be supported, but of course the FPS will be limited, making them less suitable for real time applications.
You can give a look at how we converted the tiny and base model in the hailo-whisper repo. For example, you can adapt the different scripts (export, convert_whisper_encoder, …) to add support for Whisper-small
Do you know if this example is already supposed to work for Hailo-10H, or not yet? Trying it now with 10H yields:
[HailoRT] [error] CHECK failed - Failed to create vdevice. there are not enough free devices. requested: 1, found: 0
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_PHYSICAL_DEVICES(74)
The check_installed_packaged.sh script shows “OK” for everything, so I’m not sure what else I can do to get this working (driver and HailoRT was built and installed from source as I don’t think there are binaries for 10H available yet)
Hi @Kieran_Coulter,
The example works with Hailo-10H as well, but the required version for that platform is 5.x, not 4.x (which is for Hailo-8/8L). For example, you can install version 5.1.0 to test the application. Apologies for the inconvenience, we will update the documentation accordingly.
Anyway, since you have Hailo-10H, we strongly recommend to move to the Speech-To-Text API, which is integrated in HailoRT and allows you to use optimized models from the Model Zoo.
- C++ example, requires version 5.1.0 or successive
- Python example, requires version 5.1.1 or successive
Thanks Pierre. I have updated to the latest version but still get the same error.
I’m happy to move to the STT API, but I can’t find a speech recognition model in HEF format in the model zoo. Should I make my own instead? I imagine I will just need to convert a Whisper model to HEF format.
I see now, there is a hailo-whisper repository a few replies above with the instructions to convert the models ourselves.
One last question. I noticed this comment in the Conversion README:
”Since the embedding operators have been removed from the model at conversion time, they must run on the host CPU.”
Is this really the case - Whisper (as an ONNX or a HEF) cannot run on the NPU? This is a potential showstopper. If it’s true I won’t bother continuing with the conversion. I am curious why this would be the case.
Let me clarify the options:
-
The speech_recognition application was created for Hailo-8 and then made compatible for Hailo-10H. The models used in this application were created with the hailo-whisper repo. The decoder embeddings were removed at conversion time for compatibility with the Hailo-8 architecture, and must be run on the host CPU. All the other operators run on the accelerator.
Running the embeddings on the host CPU does not add much overhead. -
For Hailo-10H only, we released other models in the Model Zoo Gen AI. These models can be used only with the Speech-To-Text API (and the models from the application example will not run with this API ).
When using the Speech-To-Text API, operations like tokenization and embeddings will be offloaded to the Hailo-10H completely.
The Whisper-Small-genai.hef from the GenAI Model Zoo v5.2.0 ( hailo_model_zoo_genai/docs/MODELS.rst at v5.2.0 · hailo-ai/hailo_model_zoo_genai · GitHub ) does not reliably respect the language parameter in the Speech2Text API. Even with language=“de” and task=Speech2TextTask.TRANSCRIBE, the model frequently outputs English instead of German. The Whisper-Base-genai.hef does NOT have this issue.
Environment
- Hardware: Raspberry Pi 5, 8GB + AI HAT+ 2 (Hailo-10H)
- HailoRT: 5.2.0
- Python: 3.13
- HEF files: Downloaded from hailo.ai/developer-zone (GenAI Model Zoo 5.2.0)
- Whisper-Base-genai.hef (131 MB) — works correctly
- Whisper-Small-genai.hef (388 MB) — language bug
- OS: Debian 13 (Trixie), aarch64
Reproduction
import numpy as np
from hailo_platform import VDevice
from hailo_platform.genai import Speech2Text, Speech2TextTask
vdevice = VDevice()
s2t = Speech2Text(vdevice, “/path/to/Whisper-Small-genai.hef”)
# audio_16k: German speech as float32, 16kHz, mono
text = s2t.generate_all_text(
audio_data=audio_16k,
task=Speech2TextTask.TRANSCRIBE,
language=“de”,
timeout_ms=30000,
)
Often outputs English instead of German!
Test Results
Test audio: German TTS (Piper de_DE-thorsten-medium), resampled to 16kHz float32 mono.
Whisper-Small GenAI (language=“de”, task=TRANSCRIBE):
Input: “Ich moechte gerne einen Termin vereinbaren.”
Output: “I would like to have a Termin Vereinbarer.” — English/mixed, WRONG
Input: “Wann haben Sie geoeffnet?”
Output: “When did they open?” — English, WRONG
Input: “Koennen Sie mich bitte zurueckrufen?”
Output: “koenne Sie mich bitte zurueckruven?” — German but with errors, PARTIALLY OK
With task=TRANSLATE all outputs are English (expected).
Whisper-Base GenAI (same API, same parameters) — works correctly:
Input: “Ich moechte gerne einen Termin vereinbaren.”
Output: “Ich moechte gerne einen Termin vereinbaren.” — CORRECT
Input: “Wann haben Sie geoeffnet?”
Output: “Wann haben Sie geoeffnet?” — CORRECT
Whisper-Base correctly respects language=“de” every time.
Analysis
According to MODELS.rst ( hailo_model_zoo_genai/docs/MODELS.rst at v5.2.0 · hailo-ai/hailo_model_zoo_genai · GitHub ), both HEFs are compiled from the multilingual HuggingFace models (openai/whisper-base and openai/whisper-small), not the .en variants.
Since the Speech2Text API offloads “tokenization and embeddings to the Hailo-10H completely” (as noted in the forum), the language token is processed on the NPU. The Whisper-Small quantization/compilation appears to have broken the language token handling, causing the model to default to English regardless of the language parameter.
Expected Behavior
language=“de” with task=Speech2TextTask.TRANSCRIBE should consistently produce German transcription, as it does with Whisper-Base GenAI.
Current Imperfect Workaround 
Using Whisper-Base-genai.hef instead. On 8kHz telephone audio we measure:
- WER: 15.7% (German)
- Latency: 424ms average
- language=“de” works reliably
I have seen the same problem and thought, the model was not able handle German.
I am currently using whisper.cpp, as the Hailo10h solution does not work sufficiently.
@Hailo team: Can you please take a look on that issue & maybe also provide the Whisper-Small HEF and/or Whisper-medium HEF (multilingual) at some point, if these are still suitable for the Raspberry PI HAT+ 2?
Hi @HerrB92 ,
Thanks for your valuable feedback.
- Whisper medium - we are working on it.
- Issue with Whisper small - We will look into it.
Thanks,
Hi @HerrB92,
We can confirm our Whisper small HEF as is worked well with German.
Can you please share the code you used for testing?
Thanks,
it was easier than I remembered (and @user540 already posted some tests above as well):
- Installed Hailo Apps (the version before the current one)
- Updated to HailoRT 5.2.0 and Model Zoo 5.2.0 (as described here)
- Downloaded Whisper-Small.hef and changed the model in the config file (as described here)
- Used a slightly modified version of
hailo-apps/hailo_apps/python/gen_ai_apps/voice_assistant/voice_assistant.py(without LLM):
The changed part in voice_assistant.py (LLM not actually used):
def on_audio_ready(self, audio):
self.abort_event.clear()
# 1. Transcribe
user_text = self.s2t.transcribe(audio, language="de")
if not user_text:
print("No speech detected.")
return
print(f"\nYou: {user_text}")
# 2. Output directly via TTS (no LLM)
if self.tts:
self.tts.clear_interruption()
self.tts.queue_text(user_text.strip())
# 3. Handshake: wait until TTS is finished, then listen again
if self.interaction:
try:
self.interaction.restart_after_tts()
except Exception:
pass
These are the test results (menu output such as “Press SPACE to start/stop recording.” omitted):
python -m hailo_apps.python.gen_ai_apps.voice_assistant.voice_assistant
2026-04-23 13:21:07.779126749 [W:onnxruntime:Default, device_discovery.cc:325 DiscoverDevicesForPlatform] GPU device discovery failed: device_discovery.cc:92 ReadFileContents Failed to open file: "/sys/class/drm/card1/device/vendor"
Initializing AI components... (This might take a moment)
INFO | common.core | Using default model: Whisper-Small
INFO | common.core | Found HEF in resources: /usr/local/hailo/resources/models/hailo10h/Whisper-Small.hef
INFO | common.core | Found HEF in resources: /usr/local/hailo/resources/models/hailo10h/Qwen2.5-1.5B-Instruct.hef
✅ AI components ready!
==================================================
Voice Assistant
==================================================
...
INFO | voice_processing.audio_player | Audio output stream started.
🔴 Recording started. Press SPACE to stop.
Processing... Please wait.
You: Mal sehen, was jetzt passiert. Ich bin sehr gespannt. This is a test. Dies ist ein Test Understood.
Press SPACE to start recording.
...
Processing... Please wait.
You: Mal sehen, was jetzt passiert. I am very curious. This is a test
In both cases, the same sentence in German was used:
“Mal sehen, was jetzt passiert. Ich bin sehr gespannt. Dies ist ein Test.”
(Translation: “Let’s see what happens. I am very curious. This is a test.”)
- “Dies ist ein Test” always becomes “This is a test”. In one case, it was even detected twice, once in German and once in English, with an added “Understood” (which was not spoken).
- “Ich bin sehr gespannt” was detected once correctly and once transcribed into English.
The result was similar for other texts in previous tests as well: a mixture of languages and sometimes also word fragments (for example, “curiou” instead of “curious”).
The microphone is fine: I tested 7 additional STT solutions with the same one. 
Hi @HerrB92,
We inspected once again and can’t reproduce.
From what we see Whisper small HEF works well with German.
Can you please try this example? hailort/hailort/libhailort/examples/genai/speech2text_example at master · hailo-ai/hailort · GitHub
Thanks,
Hello,
We would like to share some feedback regarding our tests with the following repository:
We tested it on a Raspberry Pi 5 with a Hailo-10H. Unfortunately, the results were not very conclusive for our use case.
We are testing with French as the input language (fr). Our test audio file was normalized and converted to the format expected by Whisper, as documented.
The HEF models currently provided are not usable for our use cases. We were hoping to be able to run a large-v3 or large-v3-turbo model with the Hailo-10H.
We also tried using the Python toolchain for model conversion from this repository:
https://github.com/hailocs/hailo-whisper
However, this repository does not seem to be usable in its current state, and the conversion process goes a bit beyond our current expertise.
Do you have an up-to-date procedure and/or repository for model conversion that we could test?
Are there any plans to provide large Whisper models for the Hailo-10H? If so, do you have an estimated timeline?
Thank you in advance for your help.
Hello,
like you I try to use whisper models with the speech2text_example and the results in French are bad. IMHO there is one major problem, no specialized model by language exist excepted for English. I think that the Whisper-Small model could be good if it was specialized for each language. Multilingual models are good only when they are large.
I am able to generate encoder and decoder models based on the https://github.com/hailocs/hailo-whisper but I don’t know how to merge both models. Note that on this repository, there is a branch (test-large) to try the conversions for medium and large-v2 variants. In order to work with this repository, I use a chroot for bookworm, and just a small patch for torch.onnx.export call where dynamo=False must be provided.
The output files are in the conversion/converted/ directory. If someone knows how to adapt to have a merged HEF model for the genai API, it will be very very useful.
Here my whole procedure for the whisper base model (in french):
https://wiki.debian.org/Debootstrap
mkdir debchroot
debootstrap bookworm debchroot http://deb.debian.org/debian/
mount proc debchroot/proc -t proc
mount sysfs debchroot/sys -t sysfs
cp /etc/hosts debchroot/etc/hosts
chroot debchroot /bin/bash
Hailo Whisper
https://github.com/hailocs/hailo-whisper
https://github.com/hailo-ai/hailo_model_zoo_genai/blob/main/docs/MODELS.rst
Nécessite Python 3.11 (debootstrap de bookworm nécessaire).
apt install \
bsdextrautils \
build-essential \
ffmpeg \
git \
graphviz \
libgraphviz-dev \
libportaudio2 \
locales \
python3 \
python3-dev \
python3.11-venv \
wget
Installer les locales manquantes pour Perl si nécessaire
dpkg-reconfigure locales
Commencer l’installation de l’hailo whisper
cd /root
git clone https://github.com/hailocs/hailo-whisper.git
cd hailo-whisper
python3 setup.py
source whisper_env/bin/activate
Vous devez récupérer l’hailo dataflow chez Hailo (un compte est nécessaire)
pip install /root/hailo_dataflow_compiler-5.3.0-py3-none-linux_x86_64.whl
Appliquer ce patch pour contourner une erreur avec onnx
diff --git a/export/export_whisper_model.py b/export/export_whisper_model.py
index 0ecce2d..3e89725 100644
--- a/export/export_whisper_model.py
+++ b/export/export_whisper_model.py
@@ -104,7 +104,7 @@ def export_model(variant, input_length, decoder_sequence_length, encoder_input):
os.makedirs(tmp_folder, exist_ok=True)
tmp_encoder_onnx_path = os.path.join(tmp_folder, encoder_name + ".onnx")
encoder_onnx_path = os.path.join("./export", encoder_name + ".onnx")
- torch.onnx.export(model.encoder, torch.randn(1, encoder_input, 1, mel_frames_length).to("cpu"), tmp_encoder_onnx_path)
+ torch.onnx.export(model.encoder, torch.randn(1, encoder_input, 1, mel_frames_length).to("cpu"), tmp_encoder_onnx_path, dynamo=False)
model_onnx = onnx.load(tmp_encoder_onnx_path)
model_simp, check = simplify(model_onnx)
onnx.save(model_simp,
@@ -143,6 +143,7 @@ def export_model(variant, input_length, decoder_sequence_length, encoder_input):
opset_version=13,
input_names=["decoder_input_ids", "encoder_hidden_states"],
output_names=["logits"],
+ dynamo=False
)
logger.info(f"Decoder exported to {decoder_onnx_path}")
Démarrer une conversion
python3 -m export.export_whisper_model --variant base
Encodeur
python3 -m conversion.convert_whisper_encoder export/base-whisper-encoder-5s.onnx --variant base --hw-arch hailo10h
Décodeur
python3 -m conversion.create_decoder_calib_set \
--encoder export/base-whisper-encoder-5s.onnx \
--decoder export/base-whisper-decoder-5s-seq-24.onnx \
--variant base
python3 -m conversion.convert_whisper_decoder export/tiny-whisper-decoder-5s-seq-24.onnx --variant base --hw-arch hailo10h
1 Like
Hello Hailo team,
We are seeing a very similar issue with Korean transcription using the official Whisper-Small HEF on Hailo-10H.
Environment
- Hardware: Hailo-10H
- OS: Windows 11
- HailoRT / firmware: 5.3.2
- Python: 3.11
- Official model: Whisper-Small.hef downloaded from the Hailo Model Explorer
- HEF compiler version: 5.3.0
- HEF SHA-256:
48161d62f788329a200eb6a94f6a241623f49267f20d749f5c3495949a8b4470 - Input audio: 16 kHz, mono, PCM16 WAV converted to writable C-contiguous float32
- API:
hailo_platform.genai.Speech2Text.generate_all_segments()
The issue is reproduced while bypassing our application code and calling the Hailo Speech2Text API directly.
Parameters tested
- task=Speech2TextTask.TRANSCRIBE
- language=“ko”
- language=None
- repetition_penalty=1.0
- repetition_penalty=1.1
The language and repetition-penalty combinations produced identical results.
Examples
-
Spoken Korean:
“한 잔 주세요”
Expected Korean transcription:
“한 잔 주세요”
Hailo Whisper-Small output:
“Give me a glass, please.” -
Spoken Korean:
“아메리카노 주세요”
Expected Korean transcription:
“아메리카노 주세요”
Hailo Whisper-Small output:
“American, please.” -
Longer mixed cafe-order utterance:
“안녕하세요. 아메리카노 한 잔 주세요. 카드 결제할게요. 테이크아웃 할 거예요.”Hailo Whisper-Small TRANSCRIBE output:
“안녕하세요. Americano 한잔 주세요. 카드 결제할게요. Take out 할 거예요”
For comparison, the official OpenAI multilingual Whisper-Small model running locally on CPU transcribed the exact same WAV files correctly in Korean:
- “한잔 주세요”
- “아메리카노 주세요”
- “안녕하세요. 아메리카노 한잔 주세요. 카드 결제할게요. 테이크아웃 할 거예요.”
We also re-downloaded the current official Whisper-Small HEF and confirmed that its SHA-256 is identical to the file used on the device, so an outdated or corrupted HEF is unlikely.
Could you please confirm:
- Whether this Korean language-token behavior can be reproduced with the official Whisper-Small HEF and HailoRT 5.3.2.
- Whether there is a corrected Whisper-Small HEF, a HailoRT update, or a recommended Speech2Text invocation for multilingual transcription.
- Whether you would like us to provide the minimal standalone Python reproduction script and a synthetic TTS WAV sample.
We can provide the complete direct-API diagnostic output as well.
Thank you.
How many months expected to have whisper medium possible to test?