Hi everyone,
I’d like to share hailo-10h-llm-compiler, an open-source (MIT) project that compiles your own Hugging Face LLM into a self-contained HEF for the Hailo-10H, served with hailo-ollama — no model-zoo HEF required.
It drives the Dataflow Compiler’s LLM flow (set_kv_cache_global_params, prefill + token-by-token network groups, KV-cache quantization) as a six-step pipeline: ONNX export → HAR parse → graph surgery → INT4/INT8 quantization → HEF compile → registration into hailo-ollama’s model store. The compiler itself is not redistributed — you obtain it from Hailo’s developer portal; the repo provides every script around it. Along the way it documents the runtime contract that isn’t publicly specified — embedded embeddings/tokenizer/RoPE resources, hailo-config.json keys, wire input encodings — and five distinct incompatibilities between a vanilla HF export and what the 10H LLM stack expects, each with its fix. HEF audit and diagnostics tools are included.
Status: experimental but functional. Validated end-to-end on a 25M-parameter LLaMA2-style model; prefill inference matches the float32 reference almost exactly. Multi-token generation through the KV-cache path still degrades — one open issue remains, fully written up with everything tried so far.
Help is very welcome, especially from anyone with insight into __tbt cache reads. The docs cover device setup, porting other models, troubleshooting, and every finding in detail.