HailoMuxer why?

General
1

I am trying to comprehend why we need a hailomuxer.

As I read hailomuxer in the doc (HERE) It says that is designed when we want a multidevice app, this means 2 or more hailo chips right?

Anyway, in this pipeline about multidevice, I don’t see hailomuxer being used (MultiDevicePipeline), and also in the simplest pipeline, I don’t see it either (SingleNetwork). However, in the Raspberry Pi examples in detection, I do see it, but I don’t understand why. I was able to work without it, but I couldn’t get the bounding boxes to match the resolution of my input video. I tried to rescale multiple times with no success; only hailomuxer did the trick.

Another thing is the difference between the doc in hailomuxer vs the actual library, it this right?

DOC
Pad Templates:
SRC template: ‘src’
Availability: Always
Capabilities:
ANY

SINK template: ‘sink_%u’
Availability: On request
Capabilities:
ANY

But using gst-inspect-1.0 hailomuxer I get
Actual Library
Pad Templates:
SINK template: ‘sink’
Availability: Always
Capabilities:
ANY

SRC template: ‘src’
Availability: Always
Capabilities:
ANY

And finally if there is one thing that I really want to answer by posting this topic is where to place “hailomuxer name=mux” and why it works well in some specific parts of the pipeline and only in those parts and none in anywhere else, I tried it all.

I realize that only works before a tee branch, or before filesrc, and in any place more, really, and for me it doesn’t make any sense before you defined it and then you use it, maybe I am missing something about how this pipelines works. Here is my example

**hailomuxer name=hmux ** #HERE WORKS OR
filesrc location=resources/apu_simple_h264.mkv name=src_0 !
queue name=queue_dec264 max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
matroskademux ! h264parse ! avdec_h264 max-threads=2 ! video/x-raw, format=I420 !

tee name=branch_save
**hailomuxer name=hmux ** #HERE WORKS OR
branch_save. !
queue name=queue_original_save max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoconvert ! x264enc ! matroskamux ! filesink location=experiment2_original.mkv sync=false
**hailomuxer name=hmux ** #HERE WORKS OR
branch_save. !
queue name=queue_scale max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoscale n-threads=2 !
queue name=queue_src_convert max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
videoconvert n-threads=3 name=src_convert qos=false ! video/x-raw !
tee name=branch_helper
**hailomuxer name=hmux ** #HERE WORKS OR
branch_helper. !
queue name=bypass_queue_diego max-size-buffers=20 max-size-bytes=0 max-size-time=0 !
hmux.sink_0
**hailomuxer name=hmux ** #HERE WORKS OR
branch_helper. !
queue name=queue_hailonet max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
hailonet hef-path=/home/diego/hailo-diego/basic_pipelines/…/resources/yolov6n.hef batch-size=2 nms-score-threshold=0.3 nms-iou-threshold=0.45 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 force-writable=true !
queue name=queue_hailofilter max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
hailofilter so-path=/usr/lib/aarch64-linux-gnu/hailo/tappas//post-process/libyolo_hailortpp_post.so qos=false !
queue name=tracking max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
hailotracker name=hailo_tracker kalman-dist-thr=.5 iou-thr=.6 keep-tracked-frames=300 keep-new-frames=3 keep-lost-frames=8 !
queue name=queue_user_callback max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
identity name=identity_callback !
hmux.sink_1 hmux. !
queue name=queue_hailooverlay max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
hailooverlay !
queue name=queue_videoconvert max-size-buffers=3 max-size-bytes=0 max-size-time=0 !
videoconvert n-threads=3 qos=false ! x264enc bitrate=5000 ! mp4mux ! filesink location=experiment2_output.mkv sync=false

And look the last example I was able to place hailomuxer after hmux.sink_0, for me is like using a variable before I defined it.

Summary

  • Why we need hailomuxer, and if someone has done a pipeline using detection where the resolution and aspect ratio don’t change
  • Difference in the docs
  • Where to place the hailomuxer element
2

Hey @diego

Welcome to the Hailo Community!

Purpose of HailoMuxer

The HailoMuxer element is designed for applications that use multiple devices and need to manage and synchronize streams from these devices. It is especially useful for scenarios where two similar streams must be merged into one while preserving each stream’s metadata. This functionality is essential for applications requiring synchronized processing of data from multiple sources, ensuring that all metadata from both streams is maintained and included in the combined output.

Example Pipeline for Detection with Consistent Resolution and Aspect Ratio

a typical use case would involve a GStreamer pipeline where two streams from similar sources (such as cameras with identical models and settings) are fed into the HailoMuxer. The HailoMuxer would then combine these streams into a single stream, keeping the resolution and aspect ratio consistent. This consistency is crucial to maintain the quality and integrity of the video data, which is important for subsequent processing or analysis tasks.

Placement of HailoMuxer in a GStreamer Pipeline

In a GStreamer pipeline, the HailoMuxer should be placed where the streams that need to be synchronized and merged are available. This typically occurs after the source elements and any initial processing steps but before further processing or output elements. Positioning the HailoMuxer at this stage allows it to effectively combine the streams and their metadata, ensuring that subsequent elements can process the unified stream as if it were a single source.

Example Pipeline Configuration:

[source1] -> [processing1] -> [HailoMuxer] -> [processing2] -> [sink]
[source2] -> [processing1] -> [HailoMuxer]

In this example, source1 and source2 feed into processing1 (which could be any preprocessing step). Their outputs are then directed to the HailoMuxer. The HailoMuxer combines the two streams, and the merged output undergoes further processing in processing2 before being sent to a sink for display, storage, or further analysis.

regards

1 Like