Using gstreamer with 2 different USB camera inputs and one .hef model

General
, ,
5

Hi @Paul_Siewert
This was not a very easy issue to solve but here is our attempt :slight_smile: We tested it on Raspberry Pi and two sources (video file + webcam, 2 webcams) and it works on our side, though there might be a few kinks that need to be ironed out. Please give it a try and let us know your feedback. We will work in the background to further improve it. Due to the complexity of the problem, it requires our degirum_tools package as well. Please make sure you have the latest version: 0.16.4

import degirum as dg
import degirum_tools
import degirum_tools.streams as dgstreams

inference_host_address = "@cloud"
# inference_host_address = "@local"

# choose zoo_url
zoo_url = "degirum/models_hailort"
# zoo_url = "../models"

# set token
token = degirum_tools.get_token()
# token = '' # leave empty for local inference

# Define the configurations for video file and webcam
configurations = [
    {
        "model_name": "yolov8n_relu6_coco--640x640_quant_hailort_hailo8_1",
        "source": "../assets/Traffic.mp4",  # Video file
        "display_name": "Traffic Camera",
    },
    {
        "model_name": "yolov8n_relu6_face--640x640_quant_hailort_hailo8_1",
        "source": 1,  # Webcam index
        "display_name": "Webcam Feed",
    },
]

# load models
models = [
    dg.load_model(cfg["model_name"], inference_host_address, zoo_url, token)
    for cfg in configurations
]

# define gizmos
sources = [dgstreams.VideoSourceGizmo(cfg["source"]) for cfg in configurations]
detectors = [dgstreams.AiSimpleGizmo(model) for model in models]
display = dgstreams.VideoDisplayGizmo(
    [cfg["display_name"] for cfg in configurations], show_ai_overlay=True, show_fps=True
)

# create pipeline
pipeline = (
    (source >> detector for source, detector in zip(sources, detectors)),
    (detector >> display[di] for di, detector in enumerate(detectors)),
)

# start composition
dgstreams.Composition(*pipeline).start()

We will publish a user guide on what these functions mean and do (but it will take a while as we have more basic guides to finish before coming to this advanced topic).

Looking forward to your feedback.

2 Likes