Is there a documented flow for compiling a custom/fine-tuned LLM to the Hailo-10H LLM pipeline?

General Discussion
1

Hi all!

I’m interested in running a custom or fine-tuned LLM on the Hailo-10H (AI HAT+ 2), beyond the models currently shipped in the hailo-rpi5-examples LLM pipeline.

Specifically, is there a documented way to compile a fine-tune of one of the existing architectures (e.g. Llama 3.2 1B) to .hef, or is the LLM pipeline currently limited to the three pre-converted models? Any guidance would be appreciated. Thanks!

2

Welcome to the Hailo Community!

At the moment, it is not possible to compile your own LLM for Hailo. However, we do support LoRA, which allows you to add domain-specific knowledge to a supported model.

The Hailo AI Software Suite Docker includes a LoRA tutorial. You can start the Jupyter Notebook server, which contains notebooks covering each step of the model conversion workflow, including a dedicated LoRA notebook, by running:

hailo tutorial