Gstreamer with Opencv support?

General Discussion
1

I have been running the basic_pipelines/detection_pipeline.py using own trained yolov8n model successfully.

I wanted to integrate cameraundistort and perspective from opencv, started with undistortion. This is how I adapted the code:

def get_pipeline_string(self):
# Camera calibration parameters
camera_matrix = “700,0,640,0,700,360,0,0,1” # Example values
distortion_coeffs = “-0.3,0.1,0,0,0” # Example values

    # Modify the source pipeline to include undistortion
    source_pipeline = f"""
        {SOURCE_PIPELINE(self.video_source)} ! 
        videoconvert ! 
        cameraundistort camera-matrix=\"{camera_matrix}\" distortion-coefficients=\"{distortion_coeffs}\"
    """
    
    detection_pipeline = INFERENCE_PIPELINE(
        hef_path=self.hef_path,
        post_process_so=self.post_process_so,
        batch_size=self.batch_size,
        config_json=self.labels_json,
        additional_params=self.thresholds_str)
    user_callback_pipeline = USER_CALLBACK_PIPELINE()
    display_pipeline = DISPLAY_PIPELINE(video_sink=self.video_sink, sync=self.sync, show_fps=self.show_fps)
    pipeline_string = (
        f'{source_pipeline} '
        f'{detection_pipeline} ! '
        f'{user_callback_pipeline} ! '
        f'{display_pipeline}'
    )
    print(pipeline_string)
    return pipeline_string

I get following error message:
gst_parse_error: no element “cameraundistort” (1)

        filesrc location="/home/pi/hailo-testJoFe/output.mp4" name=source ! queue name=source_queue_dec264 leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! qtdemux ! h264parse ! avdec_h264 max-threads=2 !  queue name=source_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=source_videoscale n-threads=2 ! queue name=source_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert n-threads=3 name=source_convert qos=false ! video/x-raw, format=RGB, pixel-aspect-ratio=1/1 !  
        videoconvert ! 
        cameraundistort camera-matrix="700,0,640,0,700,360,0,0,1" distortion-coefficients="-0.3,0.1,0,0,0"
     queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0

I found out:
gst-inspect-1.0 --plugin opencv
No such element or plugin ‘opencv’

I tried to compile opencv, gtreamer, … but without success.

My question(s):
Is opencv supported within Gstreamer?
If yes, how to get it working?

Thanks for your support :slight_smile:

2

Hi @johnfee welcome to our community.
Gstreamer does not come with opencv support as default. You may need to install it manually or compile from source.
See Gstreamer documentation for support.

3

Thank you for the fast reply.

Which versions of Gstreamer and opencv do I need to compile?
… I think here is my problem.

5

I am using the latest hailo SW software from January 2025 on Raspi.

6

I hope it’s OK to continue a thread from January 2025…

I am also trying to correct perspective and other distortions of the webcam view(s) in the pipeline. According to the thread above, gstreamer and opencv cannot do this and the suggestion was to compile from source.

Has anyone succeeded with image correction in the pipeline, or in any other way on a RPi? I could use OBS but it’s quite resource-hungry and makes the fan spin a lot.

Compiling from source would be very new to me, so if that’s the only way then please guide me a bit.

Thanks,

Gabor

7

Opencv is a plugin for GStreamer and Opencv contains cameracalibrate and cameraundistort.

Here is how to install GStreamer including the gstreamer1.0-plugins-bad at https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c

Would it work on the RPi 5 (with sudo) and would it not mess up the existing GStreamer? Or is there a specific version for RPi Trixie?

Install GStreamer on Ubuntu or Debian

Run the following command:

apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

Thanks,

Gabor

8

Hi @Gabor_Barton,

One solution I can think about is to apply cv2.undistort() in the user callback of your Hailo pipeline app. You get a frame as a numpy array there, so you can undistort it with a couple of lines of OpenCV Python - no GStreamer plugin needed.

Please let me know if this was helpful.

Thanks,

9

Hi @Michael,

Thanks for your help.

I think your suggestion would undistort the video after the pose estimation. I need to undistort the video before the pose is estimated, hence the need to add undistort to the source_pipeline.

Gabor

10

Hi @Gabor_Barton,

The fix is to attach a GStreamer pad probe before inference that modifies the frame in-place using OpenCV. No extra plugins needed:

import cv2
import numpy as np
from gi.repository import Gst

# Precompute undistort maps once (adjust matrix/coeffs/resolution to your camera)
camera_matrix = np.array([[700, 0, 640], [0, 700, 360], [0, 0, 1]], dtype=np.float64)
dist_coeffs = np.array([-0.3, 0.1, 0, 0, 0], dtype=np.float64)
map1, map2 = cv2.initUndistortRectifyMap(camera_matrix, dist_coeffs, None, camera_matrix, (1280, 720), cv2.CV_16SC2)

def undistort_probe(pad, info):
    buffer = info.get_buffer().make_writable()
    success, map_info = buffer.map(Gst.MapFlags.WRITE)
    if not success:
        return Gst.PadProbeReturn.OK
    caps = pad.get_current_caps().get_structure(0)
    w, h = caps.get_value("width"), caps.get_value("height")
    frame = np.ndarray((h, w, 3), dtype=np.uint8, buffer=map_info.data)
    cv2.remap(frame, map1, map2, cv2.INTER_LINEAR, dst=frame)
    buffer.unmap(map_info)
    return Gst.PadProbeReturn.OK

After your pipeline is created, attach it to the pad just before inference:

element = self.pipeline.get_by_name("inference_scale_q")
element.get_static_pad("sink").add_probe(Gst.PadProbeType.BUFFER, undistort_probe)

This undistorts every frame before it reaches hailonet, so your pose estimation runs on the corrected image.

11

Hi @Michael,

I want to add the undistort in pose estimation. Where exactly do I add the code? pose_estimation_pipeline.py? The imports go to the top but where do I add the precompute variables, where do I add the def undistort_probe(pad, info): and where do I add the last two element…. lines?

Alternatively, if you have a working example, I could learn from that.

Thanks,

Gabor

12

Hi @Gabor_Barton ,

import cv2
import numpy as np
from gi.repository import Gst

# Add the precompute variables and probe function right before the GStreamerPoseEstimationApp class definition

# --- Camera undistortion (adjust to YOUR camera) ---
CAMERA_MATRIX = np.array([[700, 0, 640], [0, 700, 360], [0, 0, 1]], dtype=np.float64)
DIST_COEFFS = np.array([-0.3, 0.1, 0, 0, 0], dtype=np.float64)
UNDISTORT_MAP1, UNDISTORT_MAP2 = cv2.initUndistortRectifyMap(
    CAMERA_MATRIX, DIST_COEFFS, None, CAMERA_MATRIX, (1280, 720), cv2.CV_16SC2
)

def undistort_probe(pad, info):
    buffer = info.get_buffer().make_writable()
    success, map_info = buffer.map(Gst.MapFlags.WRITE)
    if not success:
        return Gst.PadProbeReturn.OK
    caps = pad.get_current_caps().get_structure(0)
    w, h = caps.get_value("width"), caps.get_value("height")
    frame = np.ndarray((h, w, 3), dtype=np.uint8, buffer=map_info.data)
    cv2.remap(frame, UNDISTORT_MAP1, UNDISTORT_MAP2, cv2.INTER_LINEAR, dst=frame)
    buffer.unmap(map_info)
    return Gst.PadProbeReturn.OK

# Add the probe attachment - override create_pipeline inside the GStreamerPoseEstimationApp class, right after the existing get_pipeline_string method

    def create_pipeline(self):
        super().create_pipeline()
        # Attach undistort probe BEFORE inference
        element = self.pipeline.get_by_name("inference_wrapper_input_q")
        element.get_static_pad("sink").add_probe(Gst.PadProbeType.BUFFER, undistort_probe)

The probe intercepts every frame at the entrance to the inference wrapper, undistorts it in-place, and then the corrected frame flows into hailonet for pose estimation.

Replace the CAMERA_MATRIX, DIST_COEFFS, and resolution (1280, 720) with your actual camera calibration values.

Thanks,

13

@Michael,

I’ve added your code to pose_estimation_pipeline.py.

When running pose_estimation.py, I get an error in undistort_probe:

buffer = info.get_buffer().make_writable()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Buffer' object has no attribute 'make_writable'

Thanks,

Gabor

14

Hi @Gabor_Barton,

It’s possible to skip it and map the buffer directly with READ | WRITE:

def undistort_probe(pad, info):
    buffer = info.get_buffer()
    success, map_info = buffer.map(Gst.MapFlags.READ | Gst.MapFlags.WRITE)
    if not success:
        return Gst.PadProbeReturn.OK
    caps = pad.get_current_caps().get_structure(0)
    w, h = caps.get_value("width"), caps.get_value("height")
    frame = np.ndarray((h, w, 3), dtype=np.uint8, buffer=map_info.data)
    cv2.remap(frame, UNDISTORT_MAP1, UNDISTORT_MAP2, cv2.INTER_LINEAR, dst=frame)
    buffer.unmap(map_info)
    return Gst.PadProbeReturn.OK

In a linear pipeline like this the buffer typically has a single reference, so the READWRITE map succeeds and you can modify the pixels in-place. Just remove the make_writable() line and change Gst.MapFlags.WRITE to Gst.MapFlags.READ | Gst.MapFlags.WRITE.

16

Hi @Michael,

You are good!

It runs now without error messages but the video is not affected.

success, map_info = buffer.map(Gst.MapFlags.READ | Gst.MapFlags.WRITE)
print(success)# <- this prints false on the Terminal
if not success:
    return Gst.PadProbeReturn.OK

success is false, so we return from the function without doing the magic in the rest of the function.

____________

It’s great that you help us but I’d like to learn how to do all this. Any tips where I should look for training material? Just don’t tell me ChatGPT…

Thanks,

Gabor

17

@Michael,

success, map_info = buffer.map(Gst.MapFlags.READ | Gst.MapFlags.WRITE) is still no good.

If I comment out the next two lines (if not success:…), i.e. force to continue, then

frame = np.ndarray((h, w, 3), dtype=np.uint8, buffer=map_info.data)

gives TypeError: a bytes-like object is required, not ‘list’

Can you think of a way to get map_info writeable and as a bytes-like object?

Thanks,

Gabor

18

Hi @Gabor_Barton,

Thanks,

19

@Michael,

Are you suggesting that the answer to this subtle question is in the documentation? I wish you were right but I think that’s a long shot. Could you help please?

Thanks,

Gabor

20

Hi @Gabor_Barton,

Another approach is appsrc - capture frames with OpenCV, undistort, then push into the pipeline. Add this function before the class (remove any leftover probe code) in pose_estimation_pipeline.py:

def undistort_capture_thread(pipeline, video_source, width, height,
                             camera_matrix, dist_coeffs):
    appsrc = pipeline.get_by_name("undistort_source")
    appsrc.set_property("is-live", True)
    appsrc.set_property("format", Gst.Format.TIME)
    appsrc.set_property("caps", Gst.Caps.from_string(
        f"video/x-raw, format=RGB, width={width}, height={height}, "
        f"framerate=30/1, pixel-aspect-ratio=1/1"
    ))
    map1, map2 = cv2.initUndistortRectifyMap(
        camera_matrix, dist_coeffs, None, camera_matrix,
        (width, height), cv2.CV_16SC2
    )
    cap = cv2.VideoCapture(video_source)
    frame_count = 0
    duration = Gst.util_uint64_scale_int(1, Gst.SECOND, 30)
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        frame = cv2.remap(frame, map1, map2, cv2.INTER_LINEAR)
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        buf = Gst.Buffer.new_wrapped(frame.tobytes())
        buf.pts = frame_count * duration
        buf.duration = duration
        if appsrc.emit("push-buffer", buf) != Gst.FlowReturn.OK:
            break
        frame_count += 1
    cap.release()

In the class, replace get_pipeline_string and add create_pipeline. Replace the source_pipeline = SOURCE_PIPELINE(…) block in get_pipeline_string with:

        source_pipeline = (
            f"appsrc name=undistort_source is-live=true "
            f"leaky-type=downstream max-buffers=3 ! "
            f"video/x-raw, format=RGB, width={self.video_width}, "
            f"height={self.video_height}, framerate=30/1, pixel-aspect-ratio=1/1 ! "
            f"queue leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0"
        )

And add this method to the class (after get_pipeline_string):

    def create_pipeline(self):
        super().create_pipeline()
        camera_matrix = np.array([[700,0,640],[0,700,360],[0,0,1]], dtype=np.float64)
        dist_coeffs = np.array([-0.3, 0.1, 0, 0, 0], dtype=np.float64)
        t = threading.Thread(
            target=undistort_capture_thread,
            args=(self.pipeline, self.video_source,
                  self.video_width, self.video_height,
                  camera_matrix, dist_coeffs),
            daemon=True,
        )
        t.start()
        self.threads.append(t)

Replace camera_matrix and dist_coeffs with your actual calibration values.

Please let me know if this works

21

Hi @Gabor_Barton,

The first link contains training section for Hailo SDK.
The second link contains curated points that tend to be extremely useful and focused for building apps.

23

@Michael,

This solution works! Now I have to generate the correct camera_matrix and dist_coeffs.

There is no way you just typed this code quickly. Where did you get it from? I want to know what you know :slight_smile:

Thanks,

Gabor