Multiple Homeassistant addons using the same device

Community Projects
1

Hey guys, I’ve been looking at the homeassistant addon here based on the hailo whisper project, which I’ve confirmed as working, and I’m using frigate NVR addon as well.

My issue is that both processes cannot access the hailo8l device that I have concurrently. Is this something that can simply be fixed in the code, or is it a limitation of the hailo8l that only one app / process can use it at a time?

I don’t know if it matters but hailo-whisper would be running the whisper-base model, and for Frigate I’m using the yolov11l model.

My setup is a RPI5 (8GB) running HomeAssistant OS (16.1), with a Hailo8l device (AI HAT+).

Thanks!

2

Very interesting setup. I actually didn’t know that the add-ons were already so far along in terms of frigate support and Wyoming for Hailo.

To my knowledge, there isn’t a container-based solution yet for Hailo devices comparable to NVIDIA’s Container Toolkit, which allows seamless sharing between multiple Docker containers. With Hailo, you can forward the device into a container (e.g. mapping /dev/hailo0), but that only gives the container raw access — and only a single process can hold that access at a time.

Hailo’s own demo applications do support multi-model / multi-process use on the same chip. For example, in the official speech recognition example, you can run Whisper with the --multi-process-service flag, which enables running additional models in parallel on the same accelerator. However, this is designed to work within the same service context and pipeline, not across two separate containers or Home Assistant addons. As an alternative approach, you might want to experiment with Home Assistant Container instead of Home Assistant OS. With the container setup, you bring your own base Linux OS and manage services manually. The trade-off is that Home Assistant Container doesn’t support addons like the OS version does, so you’d be responsible for running and maintaining those services outside the Home Assistant ecosystem.

However, I would also be very interested in an official response from the Hailo developers.

Even though it has nothing to do with the topic, I would be very interested in your Hailo Frigate setup.

3

Hay Mathias, I came to the same conclusion, and have raised and add-on issue in the homeassistant OS repo Multiple addons able to use the same hailo device · home-assistant · Discussion #660.

I think for it to work there will need to be a dev effort in the OS to provide an orchestration layer somewhat similar to the Nvidia one you mentioned. It may even be simpler given that it could be built in python, and based on the scheduler.

Let me know what details you want for my Frigate setup, and I’ll share them happily.

4

Hi Barry,
I think raising the issue in the Home Assistant OS repo was definitely the right move. Especially with Hailo being an official accessory, it would be great if Home Assistant could adapt accordingly and provide proper orchestration out of the box.

I’ve just migrated my setup from a Raspberry Pi 4 to a Pi 5 and also bound the Hailo AI HAT+ (26 TOPS). Integrating Hailo into frigate was straightforward. I currently only have one camera connected and only a very basic setup.

I’d be really interested in your experience with Frigate:

  • Do you see high CPU usage on the Pi when running multiple cameras with FFMPEG?
  • How high is the Hailo utilization in your setup? I could imagine that if the Hailo load is already close to its limits, even parallel usage across addons (and Multiple camera streams ) might not leave enough headroom for Frigate or Whisper to keep up. And if Whisper with Wyoming can’t reliably recognize speech in real time, one would probably end up moving that workload to another Pi with Hailo anyway.

Looking forward to hearing about your setup!

Best regards,
Mathias

Detector Usage Frigate
Detector Usage Frigate1599×322 23.2 KB

5

Hey Mathias, so I’m running the Hailo 8L (13 TOPS) on 4 cameras (DS-2CD1383G2-LIUF/SL - Network Cameras - Hikvision Global).

I tried a number of different setups, to optimize for my use case (which is face detection in near-ground, but still getting car detection on mid / far-ground).

Initially I ran the default yolo model in frigate with my detect stream running at 720p, and 4k on my recording / audio stream. I was getting inference times ~17ms, CPU was never above 20%, and RAM stayed stable ~45% (expected as I’m running several other addons as well). However I found that I wasn’t getting the detections that I was hoping for on mid / far ground. So I tried several of the yolov11 models (the l, m, n, s, and x variants). Inference times went up accordingly and I found that the l variant gave me the best detections with an acceptable inference time ~75ms.

I was now getting good detections on near / mid-ground, but nothing further. Also face recognition wasn’t really picking up anything.

So next I just threw everything at it and gave it the 4k streams for the detect role. As expected RAM and CPU went through the roof (both 95% constantly) with 80%+ ffmpeg use for each camera.

This basically made frigate unusable, as detections weren’t happening most of the time due to the sheer volume of frames it was trying to sort through.

So I tinkered with it until I settled on setting the main stream of the cameras to roughly 2k (2688 * 1520, this was an inbuilt option, why it wasn’t 2k is beyond me).

I’m now getting detections near / mid / far (which are ridiculously accurate), as well as face detection on near ground (pretty accurate ~80%), and lpr on near ground.

The trade off is that my Pi5 is running at ~70% CPU and ~80% RAM usage pretty constantly. I’m considering buying another one to run homeassistant on, and dedicating the one with the Hailo device to Frigate (but this then defeats the purpose of the initial post).

The Hailo8l device itself is not nearly strained (as below). It’s being used, but there’s no doubt that it could handle at least another 2 processes of the same magnitude with little stress.

image
image1543×310 18 KB

The cameras are showing high ffmpeg usage, but I haven’t noticed that it’s affecting performance.

image
image1567×840 90.1 KB

And for reference, these are some of the detections.

  • Person near ground

image
image388×448 34 KB

  • Car far ground (~85m)

image
image384×443 37.6 KB

  • Car near / mid ground (~10m)

image
image380×440 48.6 KB

  • Facial recognition near ground (2-5m)

image
image599×310 35.7 KB

6

Hey Barry,

Wow, thank you so much for that incredibly detailed and comprehensive write-up! Your findings are super insightful and genuinely helpful.

Your post actually helped me pinpoint the bottleneck in my own setup. The key difference seems to be the video codec. My cameras use H.264, and unfortunately, the Raspberry Pi 5 lacks hardware decoding for it. This is why my CPU usage is already at ~20% for a single 720p stream—it’s all being processed by the CPU. Your Hikvision cameras are using H.265 (HEVC), which the Pi 5 can decode in hardware. That’s a complete game-changer and explains how you’re able to manage four high-resolution streams so effectively.

Honestly, it’s seriously impressive that you’ve achieved such accurate, multi-distance detections on a Pi 5. Pushing it to 70% CPU is a testament to how much you’re getting out of that little board!

You’ve actually inspired me to rethink my approach. I’m now considering moving my Hailo-8L to a different ARM64 board that has better hardware decoding support, as I don’t think I can get the performance I need from the Pi with my current cameras. I definitely won’t be using the 26 TOPS version; as you’ve shown, it would be massive overkill for my needs.

Looking at your Hailo utilization graph, it seems pretty clear that the accelerator itself is barely breaking a sweat. From the Hailo’s perspective, running Whisper in parallel shouldn’t be a problem at all. I believe the Whisper implementation relies on the host’s CPU and RAM for preprocessing , so with your memory usage already around 80%, that could be a limiting factor. However, before considering upgrading to a Pi with more Ram, it would first be necessary that mutiple Homeassitant Addons could use the Hailo Device.

Thanks again for sharing all your testing and results. It’s been extremely valuable!

Best regards,

Mathias

7

No problem at all Mathias, I’m glad it helped!

Yes, the usage of RAM and CPU, may well necessitate another board for me, but I’ve been more than pleasantly surprised with the performance of the Hailo 8L itself.

I’ll have a look at frigate further to see if there’s any further offloading to GPU, or the hailo device I can perhaps do.

However, since starting the conversation here, I’m definitely getting he feeling that the solution needs to be in HomeAssistant rather than a Hailo solution.

That being said, if the devs here were maybe able to give a bit of insight as to whether there’s anything on the horizon for the problem here or not, that would be appreciated.

8

Hi guys,

Regarding supporting multiple add ons in Home assistant. It is possible to run from multiple dockers, but the current ‘inference mode’ used in Frigate is not using the “multiprocess service”. So Frigate is “locking” Hailo and not allowing it to be used in additional addons.
To support that we need to enable the multi process service and make sure all addons use it.

We will try to test and push this to Frigate. Aligning the community will take time but with your help we will reach it eventually.

Very happy to see this integration to HA and Frigate ramping up :smiley:

9

So I understand correctly @giladn: Are you saying that there is no need for anything to change in HomeAssistant OS, as multiple docker instances (addons in this case) should be able to access the hailo device simultaneously as long as the caller (Frigate in this instance) is using the multi-process service?

In that case, as soon as the change is introduced to Frigate, it should enable other addons to access the hailo device as well, however the conversation here seems to imply that this is not currently possible Allow other addons to use the same Hailo device (Hailo8l in my case) · Issue #19504 · blakeblackshear/frigate.

I hope that this is a misunderstanding, or are there any specific driver versions that need to be used?

BTW using Frigate with the hailo device hs been fantastic since I implemented it. I can’t believe how accurate the car / person detection has been!

Thanks!