Is it possible to swap out the Hailo 8L that comes with the Raspberry Pi AI Kit for a Hailo 8 in order to achieve more performance?
1 Like
If it it is a Hailo-8 in the M-key format (which should also be the 2242 size), then yes. I have a M.2 M-key hat for my raspberry pi, and I purchased the Hailo-8 (26 TOPS) separately and inserted it onto the hat and it works perfectly fine.
But to be honest, unless money is no concern, I’m not quite sure it’s worth it paying the extra money for the Hailo-8.
13 TOPS in the Hailo-8L is quite a LOT of processing power to add to the Pi, and 26 TOPS is a bit overkill. The Hailo-8 M-Key 2242 I got uses a PCIE3.0x4 connection, and I doubt that anything one might do on the Pi (which uses a PCIE3.0x1 connection) would fully saturate the Hailo-8.
I’m finding that the Hailo-8 (26 TOPS) is better utilized in my AMD mini-PC, where I can have it connected via a PCIE4.0x4 connection (downgraded to PCIE 3.0 of course) and can experiment with running MANY batches of smaller neural networks (such as object detectors) simultaneously.
But if I understand correctly, when it comes to the 8L (13 tops) vs 8 (26 tops), it’s not that the 8 is faster than the the 8L, but rather it runs at the same speed.
Instead, it has twice as many compute units which allows it to run (theoretically) twice as many instances of the same neural network — or perhaps twice as many different neural networks simultaneously.
But given the constraints of a lack of DRAM on the Hailo-8(L) cards, (something that will be solved by the Hailo-10H), you are constrained to running smaller networks that are sub 100-200m parameters, which mostly entail things like CNNs, simpler vision transformers, and perhaps smaller Whisper models soon.
If you want to run more advanced stuff — and run it faster — the lack of DRAM and the needed shuffle data back and forth across PCIE 3.0x1 (or x2 or x4 depending on your Hailo module and your system) is a major constraint that will be solved by the Hailo-10H.
It feels more worth it to spend $70 on a Hailo-8L now for accelerating / experimenting with simpler neural networks , and hold back to spend more money on a Hailo-10H when they are released.
But if I’m misunderstanding anything, hopefully one of the devs can jump in and correct the record!
Thank you, you answer was very helpful actually. By the way, where did you find the RAM (and for that matter, other specs) on the Hailo 8L? I have not been able to find them anywhere.
1 Like
I’ve searched far and wide to get the official memory specs of the Hailo-8 and the Hailo-8L, and there is no official documentation or statement from the company that confirms how much memory is available on the chip.
However, there are some research papers and industry studies out there that say that the Hailo-8 is believed to have 32mb of total SRAM on board. (Source: Table on the bottom of page 4 of this report).
To put that into perspective, another popular TPU, the Coral Edge TPU, which can provide 4 TOPS of compute in INT-8, has 8 MB of SRAM.
The Hailo-10H, which isn’t released yet, will have 8 GB of LPDDR4 DRAM on the m.2 card, and can operate at up to 50 TOPS. In terms of the DRAM, that enough to run LLM’s of up to/around 8-12B parameters (depending on the quantization scheme used) directly off the card, after loading them into DRAM once over the PCIE3.0x4 connection.
1 Like