Convert a language model from onnx to hef

General
,
1

Hi everyone,

I am attempting to test if I can run a language model, or it can even be a different kind of model, on the Raspberry Pi AI kit. I can not seem to find a good guide to do this conversion, do you know of one?

The only one I found so far was with hailo model zoo, but it only applies to a list of models, not any model.

Cheers,

2

Hey @freja.thoresen ,

Welcome to the Hailo Community!

Currently, the Hailo-8 doesn’t support LLMs directly. The Hailo-8 is specifically designed for edge AI applications, with its primary strength being computer vision tasks like object detection, image classification, and segmentation.

Let me explain why running LLMs on Hailo-8 is challenging:

The Hailo-8 isn’t ideal for LLMs because:

  • These models require substantial memory and computing power
  • The Hailo-8’s architecture and Model Zoo are optimized for vision tasks, not language processing
  • The platform has specific memory and bandwidth limitations that make it unsuitable for large models

However, here are some alternatives you might consider:

  1. You could try working with smaller, compressed transformer models like DistilBERT or TinyBERT, though this would need significant adaptation for Hailo compatibility.
  2. Consider a hybrid approach where you use:
  • Hailo-8 for handling pre/post-processing tasks
  • A separate processor (CPU/GPU) for running the actual LLM inference
  1. If you’re specifically interested in language models and more complex AI tasks, you might want to look into the Hailo-10H, which is better equipped for such workloads.

The platform may expand to support LLMs in future releases, but for now, I’d recommend focusing on vision-based applications with the Hailo-8.

Feel free to ask if you need more specific guidance!