Looking for more info to work with 2 streams and 2 networks

General
14

Hey @saurabh

1. Implementing the Callback Function with HailoPython

Below is an example of how to implement the callback function to properly handle the frame data, extract tensors, and convert them to NumPy arrays.

import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib
import hailo
import numpy as np

def get_numpy_tensor(tensor):
    """
    Convert a Hailo tensor to a numpy array.
    """
    data = np.frombuffer(tensor.data(), dtype=np.uint8, count=tensor.size())
    return data.reshape(tensor.shape())

def run(frame, **args):
    """
    Callback function invoked by the HailoPython GStreamer element.
    Processes tensors and detected objects from the frame.
    """
    print('--------CALLBACK STARTED------------------')

    # Extract ROI (Region of Interest) from the frame
    roi = frame.roi

    # Extract tensors and detected objects
    tensors = roi.get_tensors()
    objects = roi.get_objects()

    print(f"Tensors Found: {len(tensors)}, Objects Found: {len(objects)}")

    # Iterate over tensors and convert them to numpy arrays
    for tensor in tensors:
        print(f"Tensor Name: {tensor.name()}")
        print(f"Dimensions: {tensor.width()}x{tensor.height()}")
        np_tensor = get_numpy_tensor(tensor)
        print(f"Converted Tensor Data:\n{np_tensor}")

    print('+++++++++++++++++++CALLBACK ENDED++++++++++++++++++++++')
    return Gst.PadProbeReturn.OK

2. Key Changes and Clarifications

2.1 Metadata Management Issue

  • Problem: When using both HailoPython and HailoFilter in the same GStreamer pipeline, tensor metadata is removed by HailoFilter after processing, making it unavailable for subsequent elements.

  • Solution:

    • Place HailoPython before HailoFilter in the pipeline to ensure the tensors are available for processing.
    • Alternatively, if you don’t need HailoFilter, you can remove it from the pipeline to preserve tensor metadata.

2.2 Correct Pipeline Configuration

To avoid the loss of tensor metadata, structure the GStreamer pipeline as shown below:

  • Correct:
    hailopython module={module_path} function=run qos=false ! hailofilter function-name={post_function_name} so-path={postprocess_so} qos=false

  • Incorrect:
    hailofilter function-name={post_function_name} so-path={postprocess_so} qos=false ! hailopython module={module_path} function=run qos=false

By following this order, HailoPython will have access to the tensors, and HailoFilter will not interfere with metadata retrieval.

3. Handling Tensor Conversion to NumPy

The updated code extracts tensors correctly and converts them into NumPy arrays using the following logic:

data = np.frombuffer(tensor.data(), dtype=np.uint8, count=tensor.size())
np_tensor = data.reshape(tensor.shape())

This ensures you can easily manipulate tensor data for further processing or analysis.

4. Debugging Tensor Data Access and Errors

If you encounter issues such as:

  • Tensor data being unavailable
  • Errors when calling vstream_info()

Use the GST_DEBUG=5 environment variable to enable detailed logging for the GStreamer pipeline. This will provide more insight into the root cause of errors and help identify any interruptions in data flow.

GST_DEBUG=5 python your_gstreamer_script.py

I hope this helps resolve the issue. If you need any further assistance, please don’t hesitate to ask.

Best regards,
Omri