I just tested the latest HEF model, and it was disappointing.
-
Hardware: AI HAT+ 2 (Hailo-10H, v5.1.1 firmware)
-
Model: llama3.2:3b (most recently added)
-
Prompt: “say hello in five words”
-
Response time: 47s
-
Response:
Hello, how are you?
That's more than 4 words. Here's a new attempt:
Hello to you.
If that is still not enough I can try again.
Hello there.
Let me know if any of those meet your request.
Alternatively I could say
Hello
Obviously, this is not acceptable as a local personal assistant. This makes the Hailo-10H a toy, not a serious NPU. I think I might get better performance from a CPU-bound llama.cpp local model.
Hi @10basetom ,
We apologize for the inconvenience this issue has caused.
We are aware for the issue (and why we added a disclaimer for this model in the GitHub page and removed it from v5.2.0).
We are looking to improve its accuracy in future versions and re-release it once ready.
Thanks,
What would be a killer feature is a compiler tool that converts GGUF files into HEF files.
3 Likes
I mean it’s a Pi hat. The business opportunity here isn’t to sell thousands of these to businesses or consumers. It’s more of a hardware development kit so people can make hardware that includes the 10H chip in it. That being said I did exactly what you did and then did a side by side of the Llama3.2:3b on the Hailo-10H hat and on my pc running the same model. The Hailo model has a shocking amount of brain damage. I’m looking through the development forums to try to find a way to convert models to Hailo’s INT4/weird firmware format. Chief among them and of great interest is Gemma4 E2B model.
1 Like
I’m running this right now on the 16GB RAM: https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/
It does all right, but I’d prefer to stuff everything into the 10H’s 8GB. There are a lot of 8GB (quantized) models out there in GGUF format.
I mean the draw of the Hailo platform is the per-watt performance of an AI core. Running models on the PI works but at an efficiency (and heat) cost. The platform I’m building every electron matters and if I can save power to use it elsewhere (or for later in my case because it’s battery powered) is a win. Hailo power savings are real. But if the models they run aren’t useful whats the point of saving the power? I think you and I think alike there.
Exactly. My main goal was adding some intelligence to my Python cron jobs, but at this rate (painfully slow and “brain dead”), I’m better off sticking to the Python logic – at least it’s predictable.
Where your device is powered (plugged in) I believe you can run regular CPU version of Ollama and potentially ditch the Hailo-10. I have a Pi 5 with 16GB of RAM I’ll run the equivalent model against the Pi CPU and see if it’s as fast but less brain dead. (at a cost of watts pulled). Also I got the Llama3.2:3b model to run on Hailo and produce significantly better results by playing with the temperature (turned it WAY down to .3) and improving my System Prompt. ~Blake
1 Like
Hi Michael, Blake —
Blake’s temperature=0.3 finding is interesting and may point to the root cause. Lower temperature reduces sensitivity to probability distribution noise. If HailoRT’s SRAM allocator is not zeroing sectors between model loads, residual activations from prior inferences would introduce noise into the token probability distributions of subsequently loaded models. Lower temperature compensates by sharpening argmax selection and suppressing that noise — which would explain why it produces better results specifically on the Hailo HEF and not on CPU ollama running the same model.
We’ve been running the 1.5B models in a multi-model sequential RAG application on the Hailo-10H 8GB/40TOPS, and observed related behavior: answer quality varies with model sequence position within a session, inference timing shows anomalous 4x variance on identical prompts, and models produce different hallucination patterns between sessions on identical fixed-weight models.
The hypothesis is that the HailoRT SRAM allocator doesn’t zero sectors between model load events. In my experience, this class of bug is particularly elusive because the failure manifests far from the actual cause — you end up looking at model quality or HEF compilation when the problem is in the runtime memory manager. This seems to be validated by a couple of other community posts.
I’ve documented this with two test runs, timing data, and a proposed verification methodology. I have documented my findings in a write up which I am enclosing a link to the document.
After posting I enabled the HailoRT runtime log and captured direct evidence during a sequential multi-model inference session. Two models loading back to back, no destroy between them:
[2026-06-05 11:39:16] [vdevice.cpp:228] Creating vdevice — qwen2.5:1.5b
[2026-06-05 11:39:16] [llm.cpp:237] Sending 6 HEF chunks to server
[2026-06-05 11:39:21] Finished loading model 'qwen2.5:1.5b'
[2026-06-05 11:41:45] [vdevice.cpp:228] Creating vdevice — llama3.2:1b
[2026-06-05 11:41:45] [llm.cpp:237] Sending 6 HEF chunks to server
[2026-06-05 11:41:49] Finished loading model 'llama3.2:1b'
No vdevice destroy between models. No SRAM release logged. The historical log also shows this from April 26 on the same hardware:
[2026-04-26 17:13:18] [error] Failed to create vdevice. not enough free devices. requested: 1, found: 0
[2026-04-26 17:13:18] CHECK_SUCCESS failed with status=HAILO_OUT_OF_PHYSICAL_DEVICES(74)
The device was still locked from the prior model when the next one tried to acquire it.
It is my hypothesis that when each model load sends 6 HEF chunks to the NPU server, with no documented SRAM release between model loads, the SRAM state present when Model B’s chunks arrive is indeterminate — it contains whatever Model A left behind. Whether HailoRT’s chunk-write process overwrites the full SRAM footprint or writes incrementally into allocated sectors is not visible from the logs. If incremental, residual data from Model A would remain in unwritten sectors during Model B’s inference. It seems to me this residual data would not simply be ignored — it may be treated as query data by the inference engine, which would explain the timeout behavior observed even in the 1.5B models when finite deterministic prompts are provided.
Paradoxically, I also observed a positive manifestation of the same mechanism. Models run later in the sequential session produced measurably better answers than models run earlier, using terminology and factual framing that I would not expected from a 1.5B parameter model on base training alone. The 4th model in sequence consistently outperformed the 1st on identical questions. If residual SRAM data from prior models contains domain-relevant vocabulary processed by earlier models, that residual data may function as unintended context — improving answer quality for subsequently loaded models. The same underlying mechanism produces different outcomes depending on the quality of what was left behind.
The updated write-up with the log evidence is at the same link above.
— Barry Berg, Apple Valley MN
apologies, I wasn’t able to update the link with the updated document.
it is located here