High RAM Usage While Running 9 HEF Models on a Shared Hailo-8 VDevice – Need Guidance

General Discussion
,
1

Hi ,

We are facing an issue with high memory consumption and are looking for ways to reduce it.

Below are the details of our setup and the specific questions we have. If anyone has experienced a similar issue or knows how to resolve it, we would greatly appreciate your guidance and suggestions.

Here are the details of our current setup:

  • Input image resolution: 1920 × 1080 (snapshot)
  • Input to Hailo device: RGB, 640 × 640, “float32”
  • Inference pipeline: 9 Hailo networks (8 detection models + 1 corrupted-image filter) running on a single shared “VDevice” with round-robin scheduling.
  • Additional processing: CPU-side tampering detection, object tracking, and image I/O.
  • Current behavior: We are not making any API calls to the server. We only store the detected image and delete the original snapshot after processing.

We have also tried the following optimizations:

  • Set “queue_size = 2” for both “InputVStreamParams” and “OutputVStreamParams”.
  • Switched from parallel inference to sequential inference execution.
  • Optimized the inference pipeline to reduce unnecessary memory allocation.

Despite these optimizations, the application is still consuming significantly more RAM than expected.

I would appreciate your guidance on the following:

  1. Is this level of RAM usage expected for this setup?
  2. What would be considered a normal or expected RAM usage for a pipeline with 9 HEF models running on a shared Hailo-8 “VDevice”?
  3. Are there any recommended HailoRT or GStreamer settings to reduce memory consumption?
  4. Are there any best practices for running multiple HEF models with lower memory usage?
  5. Is the high memory usage more likely caused by the HEF models or by the Python/GStreamer pipeline?
  6. Are there any profiling tools you recommend to identify where the memory is being consumed?

Any suggestions or best practices would be greatly appreciated.

Thank you!

1 Like
2

Hi @Aman_Srivastav :waving_hand:

I’ve run several HEFs on a shared vdevice on a Hailo-8/8L, so here is where I would look — with the caveat that I don’t have your actual RAM numbers, so this is more “what I would check” than a diagnosis.

The first thing I would change is the float32 input. A 640×640×3 float32 buffer is around 4.9 MB, and the Hailo input layer usually takes uint8, so feeding float32 means the host keeps a buffer 4× bigger than it needs, plus an extra conversion on the host that is not needed. Sending uint8 brings it down to ~1.2 MB per frame, and with 9 models each keeping input buffers times the queue depth, that difference gets big fast.

The second one is probably bigger: check whether your HEFs run NMS on-chip or you decode the raw detection outputs on the host. If the heads output raw feature maps, each output tensor is large and the framework keeps one per model per queue slot. With on-chip NMS the output is just the boxes, which is tiny. On my side that single choice moved host memory more than anything else, it was the biggest change I saw.

About the scheduler — round-robin keeps all 9 networks configured on the vdevice at the same time, so all their host buffer pools live at once. Since you already moved to sequential inference, it is worth asking if you really need all 9 resident, or if the pipeline runs them in a fixed order and you could configure a smaller set at a time. It is a trade-off between RAM and latency, but the cost of the low switching latency is having all the pools alive.

One more thing I would question: the corrupted-image filter as a full HEF. That is a whole network (device context plus host buffers) just to gate corrupt frames, and a cheap CPU check — decode error, size, a couple of stats — might replace it and give you back one full model slot. Worth measuring at least.

And before going into any of this, I would separate device RAM from host RAM, because they are different problems. hailortcli monitor shows the device side; host RAM you profile with the usual OS tools — tracemalloc if it’s Python, or /proc//smaps and valgrind massif otherwise. More than once what looked like “Hailo RAM” on my side turned out to be OpenCV holding decoded 1080p frames, not the device at all.

What I would not expect to help much: shrinking the queues below 2 (you already did that), or looking at the 9 HEF weight blobs themselves — the weights are small next to the input/output buffer pools multiplied by queue depth and number of models.

Hope this helps, and if you can share whether the growth is on the device side or the host side, it is easier to point at the real problem ..

3

Bonjour

Pour information voici les informations concernant la ram du Hailo-8L

https://tobias-weiss.org/content/ai/gemma-4-raspberry-pi-hailo-8/

Si ca peut aider…

Encore bravo pour l’exécution de 9 prossecus en même temps avec votre hailo 8