I was wondering if the team has any plans or ways to run the new Gemma 4 E4B 4 Bit Quant on the 10H? I wish this could be run on the Pi using the Ai Hat +2.
1 Like
I agree with you, I’ve tried compiling it for Hailo 10h but haven’t been able to. I hope there’s official support soon.
1 Like
Get your gloves on and get out a shovel. There’s a lot of work to be done to get that model on the Hailo-10H. I started late last week in their developer zone and it’s clear that 1. I need more time to understand how Hailo works and how its models are “optimized” to work on the hardware. 2. I may need to be smarter than I actually am to remove the blocker #1. I’ve been playing with the Llama3.2:3b model and if we don’t understand what specifically happened to that model to make it almost completely brain dead when moving to the HEF format, converting Gemma4 E4/2B models won’t matter.
Also, read Hailo’s press releases. The business model for them is to get these chips embedded in other hardware. The consumer gadget “add-on” market is really small, look at how long it took Raspberry Pi to become a household name. I’m a fan of keeping Hailo in business and them doing what they need to do to keep the lights on. This means the DIY sector will need to help eachother to make meaningful progress.
This would be amazing. Gemma4 E4B would be perfect for the Hailo10. Would really love to put that chip to work a bit more for openClaw purposes 
1 Like
I’m interested in people using Hailo 10 for openclaw. This application implies it is connected to the internet so it can “do stuff” which implies it’s plugged in. Which implies low power 15w vs 5w (10w difference) doesn’t really matter. What is your application that highlights the ability of the Hailo? A 16GB Pi5 can do what the Hailo can do… at a higher power draw with the added benefit that you can use regular Ollama models, not wait for Hailo’s format.
1 Like
Hallo everyone,
I’m playing around a bit with Gemma (e2b). Problem with almost every model on the Hailo10 is the memory bandwith - its just above 17gb/s (same as raspberry 5). This means with 2.5GB-Model one achieves around 17.5/2.5 = 7 TPS theoretical maximum. To few for productive use.
But the hailo provides raw computing power. The NPU waits 99,8% just to get the weights. Here Speculative Tree Decoding comes into the system. Accounting just for raw NPU power, 128 (or 256) parallel token verification shouldn’t be a problem.
Points to consider:
- shared KV cache (otherwise at least for sram there would rise a huge issue - beside some added memory bandwidth problems)
- With 128 (/ 256) predictions, NGRAM is the only feasible predictor model I can think of
- Tree depth / width should adapt to job (chat [language] vs coding / agent). So the attention matrix for the total tree should be an input to the network to be flexible. Injection of weight matrices in the low level API would be an alternative option, but I don’t want to mess around with that.
- the ngram should provide attention matrix + all 128 prediction tokens.
- the ngram-lut should be calculated on the raspi cpu (as i understand, the npu is not flexible enough to do that fast)
- especially for gemma, the embedding tables (also the inner ones) should be provided by the cpu (already embedded vectors for the whole text input). This saves us some memory for the number crunching.
- as i understand, the 2 models hailo uses for the llms, one (prefill) is optimized for batch processing, one for token by token inference. since every inference is batch for my model, i don’t need a “inference model”. I’m using the same for prefill as for batch inferencing. This half’s the memory space requirements in comparison to the current hailo LLM models.
For coding and agents this should give some speedup…
Currently I’m messing around with ONNX graph-surgeon, hailo DFC and also some alternative LLM models (Qwen). But the struggles with the whole setup is real, so maybe the next DFC iterations provide a bit more helpful tools and guides for LLM compilation. In addition the actual hardware constraints would be nice to know - to know if it’s possible to implement this (sram space, possible atomistic operations, int4 / int8 capabilities, …)
What are your thoughts about this setup?
2 Likes
Can’t wait to be able to play with Gemma 4 on my AI Hat+ 2 (Hailo-10).
As far as I know, there are people already trying to use the Hailo compiler (DFC) to convert gemma4:e4b to .hef format, but they are encountering conversion errors because Gemma 4’s architecture is very different from version 3. IT seems the NPU doesn’t readily understand these new layers without a major update to Hailo’s software…
4 Likes
Bonjour
Aucune réponse de la part de l’équipe Hailo? Depuis avril ?
Je comprends pourquoi la carte est si lente avec les LLMs
Ils feraient de se bouger un peu afin que les utilisateurs puissent mettre les modèles de LLM qu’il désirent via une application simple ( et pas faut renomer ci en cela et pas oublier un argument)
Choix du modèle entré en GGUF sortie en HEF
A mettre dans le répertoire d’ollama ( pour l’instant c’est olama qui télécharge les modèles autorisés)
Informations à l’équipe de hailo on peut faire tourner Gemma 4 2B sur un Raspberry 4 avec Turboquant et openclaw
A quoi bon vouloir acheter un raspberry pi 5 avec un module accélérateur Hat+2 40tops hailo 10H ?
Si en plus Les modèles sont plus lent dessus:
Au cas où l’équipe hailo ne publierais pas mon message sachez que j’en ai fait une copie …j’attends une réponse concrète sur votre imobilimes (vous m’avez même pas fait l’effort de réponde au post précédent)
1 Like
You are right to be frustrated and so am I. This was an absolute waste of money and time. Its actually a toy to make toy projects not even fit for hobbyist at this point. Developers don’t engage in this forum because they know they will have to answer some uncomfortable questions.
2 Likes
le blocage n’est ni la RAM ni le compilateur (le Dataflow Compiler compile deja des LLM, c’est comme ca que le zoo a ete construit), c’est le runtime generatif qui refuse de charger un .hef compile hors zoo.
soit ouvrir ce runtime aux modeles compiles par la communaute.
soit ajouter Gemma au zoo officiel
sounds like your on the struggle bus, i run llms on the h8. at sub bit quant. i have 50+ chips running production deployments in commercial facilities, the chipset is more than capable. The H10 even more so. just my 2 cents, not that you asked. automatanexus.com
1 Like
Bonjour
Votre système en hailo 8 à l’air super !
Mais quel est le rapport avec Gemma 4 2B ou 4B ?
On voudrait pourvoir installer n’importe quel LLM ( avec une taille raisonnable ) dans la ram du module hailo 10,
Pour l’instant 6 modèmes LLM ou plutôt 5 car ils en supprimer 1 dans la dernière version du firmware .
Ils pourraient intégrer turboquant afin améliorer la vitesse ( quelqu’un la bien fait sur un raspberry pi 4)
Haillo ont l’air d’avoir un super marketing mais que font les ingénieurs de la R&D?
De plus le fait qui ne communique pas n’arrange rien…
I hear the frustration, so let me offer some encouragement: getting real transformer-style LLMs onto this hardware is doable — and it isn’t limited to a handful of blessed models. On the Hailo-8 I’ve been able to run a genuinely broad spread of architectures:
decoder-style transformer LLMs — attention, RoPE, GQA, RMSNorm, gated MLPs — including a 1.58-bit ternary model;
CNN detectors with attention blocks;
recurrent nets (GRU and LSTM);
temporal conv nets (TCN);
and autoencoders.
I’ve been working on the Hailo-10 side as well. So “a reasonable-size LLM living in the module’s RAM” isn’t a pipe dream — it takes patient engineering, but it’s reachable.
On the “brain-dead after conversion” problem a few of you hit (the Llama-3.2-3B experience): in my experience that’s almost always the quantization, not the chip. Post-hoc quantization can cost a model a lot of quality; models quantized training-aware, or built low-bit from the start, hold up far better. And one habit that saved me endless debugging — always compare your on-device output against a CPU/reference run before trusting it. If they line up, you know the conversion didn’t quietly break the model; if they don’t, you know exactly where to look.
On TurboQuant and fitting into RAM — that’s the right instinct. Quantizing the KV cache alongside low-bit weights is how you get a bigger model to fit and stay responsive.
Happy to trade notes with anyone chasing the same thing — how to shape a transformer for this hardware, where accuracy actually goes during quantization, KV strategies, and so on. Feel free to reach out.
— Andrew
1 Like
Bonsoir
Nous serions ravis d’en discuter
Voir mon 1er post sur TaOS
Pour Gemma 4 j’ai trouvé cela:
Gemma 4 E4B + E2B Non censuré (Agressif) — GGUF + K_P Quants (Multimodal : Vision, Vidéo, Audio) : r/LocalLLM
Ça a l’air d’une bonne base?
Good evening 
Nice project — RPi 5 + Hat 2 (Hailo-10H, 40 TOPS) is an ideal edge platform for this kind of model.
Gemma as a base: yes, a solid choice. The architecture runs well on my side — I’ve had the Gemma family running end-to-end (Gemma-style RMSNorm, GeGLU blocks, RoPE), along with Llama, Mistral, Qwen 2/3, Phi, Falcon, GPT-J, GPT-NeoX, some MoE (Mixtral, Qwen-MoE), and Mamba/SSM. So Gemma E2B/E4B is no problem architecturally, and those sizes are well-suited to the 10H’s memory.
Two practical notes:
GGUF is perfect as a weight source / reference. Just keep in mind the K-quants (K_P, etc.) are specific to llama.cpp / CPU — on the NPU you’ll start from the original weights, not the quantized .gguf as-is.
Multimodal (vision / video / audio): start with text-only to validate the whole decode path, then add the modalities one at a time. The vision side in particular fits edge really well.
The “uncensored / aggressive” part makes no technical difference — it’s just a fine-tune; what matters is the architecture, and it runs.
Happy to compare notes on LLM architectures on Hailo if it helps move the project forward 
1 Like
Bonjour
Je ne suis pas développeur de métier
Mais ça empêche pas d’aider à ma façon
Voici ce que j’ai trouvé :
https://www.reddit.com/r/LocalLLaMA/comments/1sb73rb/run_gemma4_with_turboquant_locally/?tl=fr
Je m’égare peut être merci de votre retour
Je pense qu’on aurait besoin d’un tutoriel…pour mettre tout ca en œuvre.
Do we have a timetable or statement if we can expect Gemma4 E4B ?
Thinking about cancelling the AIHat2 purchase.
1 Like
Sucks that Devs are not making any visible progress nor interacting with the community here. It was a bad purchase not very useful I must say
3 Likes
Did any one manage to get Gemma working?