How to use two cameras with Yolo modules for RPI5 with HAT+ Hailo ?

General Discussion
1

Hello, Peers! I have connected two cameras to both ports of my RPI5 with HAT+ Hailo boards and a have well tested operation of Yolo modules separately with each camera. But how can I use Yolo modules for object detection with both cameras simultaneously? I woild like to see images from both cameras on my monitor. What should I change in the code for that? Thank you!

2

Hey @Igor_Zaitsev1 ,

You can combine two USB cameras in a single GStreamer pipeline while using the Hailo framework for inference. Here’s how:

Create a composite pipeline that:

  • Takes input from two cameras (/dev/video0 and /dev/video1)
  • Combines both streams using the compositor element
  • Runs the combined output through your inference pipeline

Implementation Example ( Not Tested !)

Modify your get_pipeline_string() method in the GStreamerDetectionApp class:

def get_pipeline_string(self):
    cam0 = "/dev/video0"
    cam1 = "/dev/video1"
    width, height = 640, 480  # Resolution for each camera

    # Camera 0 pipeline
    cam0_pipeline = (
        f'v4l2src device={cam0} ! image/jpeg, width={width}, height={height}, framerate=30/1 ! '
        f'decodebin ! videoconvert ! videoscale ! '
        f'video/x-raw, width={width}, height={height} ! '
        f'queue ! compositor.sink_0 '
    )

    # Camera 1 pipeline
    cam1_pipeline = (
        f'v4l2src device={cam1} ! image/jpeg, width={width}, height={height}, framerate=30/1 ! '
        f'decodebin ! videoconvert ! videoscale ! '
        f'video/x-raw, width={width}, height={height} ! '
        f'queue ! compositor.sink_1 '
    )

    # Compositor element
    compositor = (
        f'compositor name=compositor sink_0::xpos=0 sink_1::xpos={width} ! '
        f'videoconvert ! videoscale ! '
        f'video/x-raw, width={width*2}, height={height} ! '
    )

    # Add inference and display pipelines
    inference = INFERENCE_PIPELINE(...)
    display = DISPLAY_PIPELINE(...)

    return cam0_pipeline + cam1_pipeline + compositor + inference + display

Key Considerations

  • Resolution: Ensure the combined output resolution matches your model’s input requirements
  • Performance: Keep framerates equal for better synchronization
  • Bounding Boxes: Will appear across the combined frame with no separation between cameras
  • Layout: Adjust positions with sink_0::xpos and sink_1::xpos parameters

If you prefer switching between cameras rather than combining them, replace compositor with input-selector.

We will be adding Guides and more pipeline examples in the near future !

1 Like
3

Dear Sir,
thank you so much for this detailed answer!
I will try to folow.
Best regards

4

I did it with python and run it with thonny it works grear. One camera runs yolo v8m and other yolo v6s. >30fps

8

I’m on RPi AI HAT+ Trixie with Hailo-Apps v25.12.0 and HailoRT 4.23.0.

I’ve added @omria’s code from above to pose_estimation_pipeline.py for reading/dev/video0 and /dev/video1 simultaneously, to see if this gives better performance than combining the videos with OBS (thanks to @Michael for helping with that one!).

I then modified from this:

pipeline_string = (
            f"{source_pipeline} ! "
            f"{infer_pipeline_wrapper} ! "
            f"{tracker_pipeline} ! "
            f"{user_callback_pipeline} ! "
            f"{display_pipeline}"
        )

to this:

pipeline_string = (
            #f"{source_pipeline} ! " # turned off
            f"{cam0_pipeline} ! " # added
            f"{cam1_pipeline} ! " # added
            f"{compositor} ! "    # added
            f"{infer_pipeline_wrapper} ! "
            f"{tracker_pipeline} ! "
            f"{user_callback_pipeline} ! "
            f"{display_pipeline}"
        )

but I’m getting

ERROR | gstreamer.gstreamer_app | Error creating pipeline: gst_parse_error: syntax error (0)

I spotted a ! ! in the pipeline_string, but even after fixing that I get the syntax error. Is this because omria’s code is from a year ago and I’m using the most recent Hailo-Apps v25.12.0 which has a different syntax? Or is there a genuine syntax error that I cannot see? Either way, please help build the correct pipeline.

Here is the pipeline_string that causes the error:

v4l2src device=/dev/video0 ! image/jpeg, width=640, height=480, framerate=30/1 ! decodebin ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! queue ! compositor.sink_0 ! v4l2src device=/dev/video2 ! image/jpeg, width=640, height=480, framerate=30/1 ! decodebin ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! queue ! compositor.sink_1 ! compositor name=compositor sink_0::xpos=0 sink_1::xpos=640 ! videoconvert ! videoscale ! video/x-raw, width=1280, height=480 ! queue name=inference_wrapper_input_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailocropper name=inference_wrapper_crop so-path=/usr/lib/aarch64-linux-gnu/hailo/tappas/post_processes/cropping_algorithms/libwhole_buffer.so function-name=create_crops use-letterbox=true resize-method=inter-area internal-offset=true hailoaggregator name=inference_wrapper_agg inference_wrapper_crop. ! queue name=inference_wrapper_bypass_q leaky=no max-size-buffers=20 max-size-bytes=0 max-size-time=0 ! inference_wrapper_agg.sink_0 inference_wrapper_crop. ! queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoscale name=inference_videoscale n-threads=2 qos=false ! queue name=inference_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! video/x-raw, pixel-aspect-ratio=1/1 ! videoconvert name=inference_videoconvert n-threads=2 ! queue name=inference_hailonet_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailonet name=inference_hailonet hef-path=/usr/local/hailo/resources/models/hailo8/yolov8m_pose.hef batch-size=2 vdevice-group-id=SHARED force-writable=true ! queue name=inference_hailofilter_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailofilter name=inference_hailofilter so-path=/usr/local/hailo/resources/so/libyolov8pose_postprocess.so function-name=filter_letterbox qos=false ! queue name=inference_output_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! inference_wrapper_agg.sink_1 inference_wrapper_agg. ! queue name=inference_wrapper_output_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailotracker name=hailo_tracker class-id=0 kalman-dist-thr=0.8 iou-thr=0.9 init-iou-thr=0.7 keep-new-frames=2 keep-tracked-frames=15 keep-lost-frames=2 keep-past-metadata=False qos=False ! queue name=hailo_tracker_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! queue name=identity_callback_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! identity name=identity_callback ! queue name=hailo_display_overlay_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailooverlay name=hailo_display_overlay ! queue name=hailo_display_videoconvert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoconvert name=hailo_display_videoconvert n-threads=2 qos=false ! queue name=hailo_display_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! fpsdisplaysink name=hailo_display video-sink=autovideosink sync=true text-overlay=False signal-fps-measurements=true

9

Can anyone please point me at documentation and/or examples of how to make pipelines? The syntax error I’m getting from omria’s code should be easy to fix.

These three lines at the beginning of the pipeline cause the error. What’s wrong with them?

v4l2src device=/dev/video0 ! image/jpeg, width=640, height=480, framerate=30/1 ! decodebin ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! queue ! compositor.sink_0 !

v4l2src device=/dev/video2 ! image/jpeg, width=640, height=480, framerate=30/1 ! decodebin ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! queue ! compositor.sink_1 !

compositor name=compositor sink_0::xpos=0 sink_1::xpos=640 ! videoconvert ! videoscale ! video/x-raw, width=1280, height=480 !

Thanks,

Gabor

10

Hi @GaborLJMU,

The syntax error comes from how the branches are joined. In GStreamer, when you link to a named sink pad (like compositor.sink_0), that’s the end of that branch - no need in ! after it. Each source branch should be separated by a space, not by !.

Instead:
... queue ! compositor.sink_0 ! v4l2src device=/dev/video2 ...

Should be:
... queue ! compositor.sink_0 v4l2src device=/dev/video2 ...

Same for compositor.sink_1 - no ! between it and the compositor name=compositor ... line.

Instead:

f"{cam0_pipeline} ! "
f"{cam1_pipeline} ! "
f"{compositor} ! "

Should be:

f"{cam0_pipeline} "
f"{cam1_pipeline} "
f"{compositor} ! "

In addition maybe this will be helpful: hailo-apps/hailo_apps/python/pipeline_apps/multisource at main · hailo-ai/hailo-apps · GitHub

Thanks,

11

Hi @Michael

Thanks, I knew this was simple! The syntax error is gone.

Trouble is, I now get another error:

ERROR: Hailo Cropper Input and output caps have different formats

I’ve played with the sizes of the sinks and the compositor, but no effect.

Yes, I’ve tried multisource.py but with two webcams it only runs at 15fps and the timestamps of the two poses will be different as they arrive at different times in the callback.

I’m still hoping that the compositor method may run at 30Hz. My fallback solution is the combined image of the OBS Virtual Camera which appears as /dev/video4, but that uses a lot of CPU and GPU.

Thanks,

Gabor

1 Like
14

The fix for the error ERROR: Hailo Cropper Input and output caps have different formats is that the compositor’s format must be specified as format=RGB(inserted after video/x-raw):

compositor = (
            f'compositor name=compositor sink_0::xpos=0 sink_1::xpos={width} ! '
            f'videoconvert ! videoscale ! '
            f'video/x-raw, format=RGB, width={width*2}, height={height} '
        )

Now I have the two webcam views side by side and pose estimation runs nicely on both. My next step is to apply distortion correction on the webcam views.

Gabor

1 Like
15

Thanks @GaborLJMU for sharing the solution!