Custom python Post processing function in gstreamer performance issues

[rosslote]

I’m not sure how to inspect the tensor types but when I convert them to numpy arrays they are all unit8

for tensor in video_frame.roi.get_tensors():
    print(tensor.name(), np.array(tensor).dtype)

yolov8s_bed/conv43 uint8
yolov8s_bed/conv44 uint8
yolov8s_bed/conv45 uint8
yolov8s_bed/conv57 uint8
yolov8s_bed/conv58 uint8
yolov8s_bed/conv59 uint8
yolov8s_bed/conv70 uint8
yolov8s_bed/conv71 uint8
yolov8s_bed/conv72 uint8

I managed to solve the FPS issue by reducing the max_detections.

There seems to be a lot of bad info out there are moment. I’m not sure if it’s due to changing apis or not. For example:

@omria

def get_numpy_tensor(tensor):
    """
    Convert a Hailo tensor to a numpy array.
    """
    data = np.frombuffer(tensor.data(), dtype=np.uint8, count=tensor.size())
    return data.reshape(tensor.shape())

Just gives this error:

a bytes-like object is required, not 'int'

Looks like data is just a number not an array.

Also:

@user67

calling my_array = np.array(my_tensor, copy=False), should automatically dequantize the tensor from integer format to float.

But my tensor is not dequantizing when I call this.

@shashi using the info you gave me I implemented this:

tensor_info = {
    "yolov8s_bed/conv70": {"scale": 0.1039922684431076, "zp": 168.0},
    "yolov8s_bed/conv71": {"scale": 0.003921568859368563, "zp": 0.0},
    "yolov8s_bed/conv72": {"scale": 0.0006715772324241698, "zp": 16007.0},
    "yolov8s_bed/conv57": {"scale": 0.09461547434329987, "zp": 151.0},
    "yolov8s_bed/conv58": {"scale": 0.003921568859368563, "zp": 0.0},
    "yolov8s_bed/conv59": {"scale": 0.0004394233983475715, "zp": 22882.0},
    "yolov8s_bed/conv43": {"scale": 0.15522289276123047, "zp": 133.0},
    "yolov8s_bed/conv44": {"scale": 0.003921568859368563, "zp": 0.0},
    "yolov8s_bed/conv45": {"scale": 0.00035622910945676267, "zp": 19781.0},
}

def dequantize_tensor(tensor):
    info = tensor_info[tensor.name()]
    zero_point = info['zp']
    scale = info['scale']
    return scale * (np.array(tensor, copy=False) - zero_point)

I hard coded the values for now until I figure out how to get this info dynamically.

Is this correct?

The output of my bbox and keypoints now looks like this:

bbox 243.2337546426842 191.12009224489435 467.467703175785 471.105251257167
kpts [[-362.5803538  -333.33113414]
 [-365.2881532  -328.38832571]
 [-361.84967777 -333.46007697]
 [-365.33113414 -335.13633374]]

The bbox is correct for pixel values but hailo overlay seems to want percentages so if I do this:

        xmin, ymin, w, h = [float(x)/640 for x in bbox]
        bbox = hailo.HailoBBox(xmin, ymin, w-xmin, h-ymin)

It draws the box correctly.

The keypoints are all negative now though so I need to figure that out.

Would be nice if i could just get the values back from the postprocessor as percentage values so I guess I should make some adjustments. The only part I can see which uses the image dimensions is here

        for box_distribute, kpts, stride, _ in zip(raw_boxes, raw_kpts, strides, np.arange(3)):
            shape = [int(x / stride) for x in image_dims]
            grid_x = np.arange(shape[1]) + 0.5
            grid_y = np.arange(shape[0]) + 0.5
            grid_x, grid_y = np.meshgrid(grid_x, grid_y)
            ct_row = grid_y.flatten() * stride
            ct_col = grid_x.flatten() * stride

I don’t really understand what stride means yet though.