Hailo 10 (AI HAT+ 2) – Converting ONNX LLMs to HEF & Ollama embedding support

General Discussion
,
1

Here’s the translation:

Hello, I have just purchased the AI HAT+ 2 for Raspberry Pi (so with the Hailo 10).

I have the following questions:

  • On Hugging Face there are small LLMs for text generation (≤2B parameters) in ONNX format. I haven’t found a straightforward guide to convert them to HEF format. What would be the simplest method?

  • Does Hailo Ollama support embedding models?

2

Hi @Philippe_Charriere,

The Hailo Ollama API is limited to LLM text generation (/api/chat and /api/generate endpoints).

However, if you need embeddings on Hailo hardware, the Hailo Model Zoo (GitHub - hailo-ai/hailo_model_zoo: The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment · GitHub) provides text encoder HEFs for text-image retrieval on both Hailo-8 and Hailo-10H. These run via HailoRT directly rather than through the Ollama-compatible API. For example: hailo_model_zoo/docs/public_models/HAILO8/HAILO8_text_image_retrieval.rst at master · hailo-ai/hailo_model_zoo · GitHub and hailo_model_zoo/docs/public_models/HAILO10H/HAILO10H_text_image_retrieval.rst at master · hailo-ai/hailo_model_zoo · GitHub

Regarding compiling LLMs into HEFs: For the time being, our Dataflow Compiler supports only non-GenAI models.

Thanks,

3

@Michael thank you for your answer. What’s the plan for new LLM GenAI models or the ability to compiling them ? Because right now, without this, the hat is useless for my use cases.

4

Hi @Philippe_Charriere,

For the time being there is no specific timeline I can share about GenAI compilation support.
New Hailo SW releases might (and will) add support for more and new models.

Thanks,

1 Like
5

I see, there’s my problem then! I tried converting multiple models using the compiler but non of them were compatible and I couldn’t figure out what I was doing wrong haha

Do you know if there is a specific tag(s) that I can filter on Huggingface to list only compatible models?

6

Hi,

Note that Hailo’s official pre-compiled HEF model zoo is NOT hosted on Hugging Face. The HF hailo tag is primarily used by community repos that publish HEF-ready or Hailo-optimized models. There isn’t a verified “hailo-ai” organization on HF with an official model collection at this time.

Thanks,

1 Like