Best architecture for dual simultaneous video stream inference with a 30M / 3-context semantic segmentation model on Hailo-8?

[Julius_Palsherm]

Hi everyone,

I’m looking for guidance on the recommended production architecture for two continuous, simultaneous video streams on a single Hailo-8 PCIe device using HailoRT 4.x (C++ API + VStreams).

Setup

• Chip: Hailo-8, single device over PCIe.

• Model: semantic segmentation, SegFormer-based, around 30M parameters.

• Compiled HEF: 3 contexts reported by DFC compilation.

• Input resolution: 512x512 per frame.

• Goal: run 2 independent camera streams continuously and in real time, both using the same model.

• Deployment: headless embedded system, output goes to shared memory / downstream processing.

The single-stream version works well. Now I need to extend this to two simultaneous camera inputs, each inference running continuously with minimal latency.

Approaches I tried

1. HailoRT scheduler with two configured network groups

I load the same HEF twice as two separate ConfiguredNetworkGroup instances on a VDevice, and let the scheduler time-multiplex them.

auto vdevice = VDevice::create(params);
auto hef = Hef::create("model.hef");

auto ng1 = vdevice->configure(hef.value());
auto ng2 = vdevice->configure(hef.value());

auto vstreams1 = VStreamsBuilder::create_vstreams(*ng1[0], ...);
auto vstreams2 = VStreamsBuilder::create_vstreams(*ng2[0], ...);

This seems straightforward, but I’m not sure if it is the best long-term production path.

2. Multi-network single HEF

I compiled two copies of the same model into one HEF as two named networks. My assumption was that one ConfiguredNetworkGroup containing both networks might allow better utilization or even true parallelism.

auto hef = Hef::create("dual_model.hef");
auto ng = vdevice->configure(hef.value());

auto net_infos = ng[0]->get_network_infos();
auto vs_net1 = VStreamsBuilder::create_vstreams(*ng[0], {}, FORMAT_TYPE, net_infos[0].name);
auto vs_net2 = VStreamsBuilder::create_vstreams(*ng[0], {}, FORMAT_TYPE, net_infos[1].name);

auto activated = ng[0]->activate();

I’m not sure whether this actually gives me parallel execution, or just makes the runtime switch between contexts internally.

3. TAPPAS multi-stream pipeline

I also looked at TAPPAS/GStreamer multi-stream examples using a funnel into hailonet and then streamiddemux.

4. TAPPAS parallel networks

I also considered the pattern where two separate hailonet elements are fed with tee.

My questions

1. Feasibility: Can a 30M parameter / 3-context model realistically be duplicated on a single Hailo-8 for true parallelism, or is that likely beyond the device budget?

2. Scheduler vs multi-network HEF: For two continuous cameras using the same model, is the recommended path:

   • two configured network groups with scheduler time-multiplexing,

   • or a multi-network HEF,

   • or a GStreamer/TAPPAS multi-stream setup?

3. hailonet behavior: Does hailonet internally use the same HailoRT scheduler / VDevice logic as the C++ API, meaning the GStreamer path is mainly a convenience layer rather than a different execution model?

4. Production recommendation: Given my constraints, which architecture would you recommend for production:

   • raw HailoRT C++ with VStreams,

   • TAPPAS/GStreamer,

   • or a different multi-stream pattern?

Current results with switched network groups

For reference, here’s what I’m currently getting with approach 1 (two configured network groups, scheduler time-multiplexed):

-I=====================================================================
-I- DUAL INFERENCE SUMMARY
-I=====================================================================
-I- Total wall-clock time: 5781 ms
-I- [inst0] Images: 34  Avg: 170 ms/image
-I- [inst1] Images: 34  Avg: 170 ms/image
-I- Combined throughput: 11.7627 frames/sec (68 total frames)
-I=====================================================================

Compared to single-stream inference, this is roughly a bit less than half the original frames/sec

If anyone has experience with dual camera inference on a single Hailo-8, especially with a segmentation model and multiple contexts, I’d really appreciate advice on the best architecture and whether there is a reference design I should follow.

Honestly, I’d love it if someone from Hailo itself could weigh in on this — official guidance on the “right” architecture for this kind of multi-context, multi-stream setup would save a lot of trial and error, and I suspect I’m not the only one hitting this question.

Thanks in advance!

[Klaus_Koschinsky]

Welcome to the Hailo-Community!

@Julius_Palsherm

let the scheduler time-multiplex them.

In most cases, you should use the HailoRT scheduler to manage network switching. This is the recommended and scalable solution, allowing you to add additional Hailo devices or networks as your application grows.

@Julius_Palsherm

I compiled two copies of the same model into one HEF

Compiling multiple networks into a single HEF file should only be considered if:

• All networks fit into a single Hailo-8 without requiring multiple contexts.
• You are certain that you will not need to add additional networks to the application in the future.

This approach allows you to load a single HEF and execute all networks without switching HEF files. It can be beneficial when the system has only a single PCIe lane and PCIe bandwidth is the primary bottleneck.

However, this approach is not scalable. As soon as you need to add another network, you will need to recompile the HEF or switch to the HailoRT scheduler.

Compiling the same network twice into a single HEF is almost never beneficial. For example, if a model can process 200 FPS, the scheduler can distribute that capacity across multiple streams. For example, 10 streams at approximately 20 FPS each.

@Julius_Palsherm

Can a 30M parameter / 3-context model realistically be duplicated on a single Hailo-8 for true parallelism, or is that likely beyond the device budget?

No.

A context is a part of a network. For example, layers 1–10, 11–20, and so on.

During inference, HailoRT loads the first context onto the device, processes it, returns the intermediate results to the host, loads the next context, sends the intermediate data back to the device, and repeats this process until inference is complete.

So, duplicating the network will not improve the conditions it will just double the number of contexts.

If you are processing multiple images, increasing the batch size can reduce the context-switching overhead because each context computes multiple images before switching to the next one (see tip at the end).

@Julius_Palsherm

For two continuous cameras using the same model, is the recommended path

Use the HailoRT scheduler.

@Julius_Palsherm

hailonet behavior

hailonet is simply a GStreamer element. Under the hood, it uses HailoRT, so the scheduling and execution behavior is the same.

@Julius_Palsherm

Given my constraints, which architecture would you recommend for production:

This is ultimately a design choice. Choose the architecture that best fits your application requirements. Different customers have different constraints, which is why we support multiple frameworks and programming languages. In some cases, the capabilities of the host platform will also influence the best choice.

@Julius_Palsherm

dual camera inference on a single Hailo-8, especially with a segmentation model and multiple contexts,

Try reducing the context switching overhead by using batch-size. You can test this without an application by using HailoRT CLI.

hailortcli run model.hef --batch-size 1
hailortcli run model.hef --batch-size 2

For a multi-context model the FPS should increase when using a larger batch-size. You can increase this further. However there is a limit for each model and HailoRT will warn you when you reach it e.g.

[HailoRT] [warning] Desc page size value (1024) is not optimal for performance.