Hailo-8 for attention-based object detection models

General
,
1

Hi everyone,

I have a couple of questions regarding what the Hailo-8 accelerator is optimized for. I asked them on Hailo’s website but I was sent on this forum for answers.

I recently purchased a Raspberry Pi 5 for a personal project involving LLMs. I initially thought about acquiring the AI Hat+ before realizing that it is not optimized for LLMs inference, but rather for computer vision tasks. I got this information from various forums but none explained the technical details as to why.

For another project I have at work currently, I want to run an object detection model on a micro computer, with a video stream coming from a single camera.
I was thus wondering if the Hailo-8 is optimized for any object detection model, even the ones comprising attention-based blocks (DETR for example or latest YOLOs I think).

My questions:

  • What makes the Hailo-8 accelerator not suitable for LLMs? I have read that it is due to the number of parameters (several billions) that is too big. Is it only that or the actual nature of the architecture?
  • Is the Hailo-8 optimized for all types of object detection models including attention-based ones?

I don’t know the other Hailo products so don’t hesitate to point me to other ones.

Many thanks in advance

Cheers

Antoine

2

Welcome to the Hailo Community!

The Hailo Dataflow Compiler can divide networks that do not fit into a single Hailo-8 into multiple contexts. During execution the HailoRT runtime loads these context over PCIe. Some CNNs have a few contexts and provide still good performance. LLM require one hundred or more context which will result in not so good performance while loading the host CPU.

The Hailo-10 has a DDR interface which allows it to run the contexts without the host. There have been some improvements to optimize context switching. Also Hailo-10 can execute two 4-bit operations instead of one 8-bit OP. Because LLM are so large they will use 4-bit layers even more than CNNs to improve efficiency.

Hailo-8 does support transformer based networks. It has a very flexible architecture that allows us to add new layer support via our Hailo Dataflow Compiler. To compare models I recommend you have a look at the Model Explorer in the Developer Zone.

Hailo Model Explorer

Have a look at our Model Zoo and the Explorer. Plenty of models to choose from including the latest YOLOs.

GitHub - Hailo Model Zoo - HAILO8 - Object Detection