Tensors are in different format in hailonet and Hailo async Inference callback

[saurabh]

@nina-vilela @omria
I am working with hailopython gstreamer element and I am processing yolov8_pose output in python.

I found a way to extract numpy from tensors in TAPPAS DOCS.

But the format seems is different than the format in HailoAsyncInference…

hailo-ai/Hailo-Application-Code-Examples/blob/a10db4d6799a4355dd71dc7bb1e99887757fc199/runtime/python/pose_estimation/pose_estimation.py#L130C26-L130C41

hailonet results are in b/w 0-255
And Hailo Async Callback is float values(Possitive and Negative).

And I am unable to utilize the post process steps for yolov8_pose from here… Hailo-Application-Code-Examples/runtime/python/pose_estimation/pose_estimation_utils.py at main · hailo-ai/Hailo-Application-Code-Examples · GitHub

from typing import Dict
import gi
import numpy as np
gi.require_version('Gst', '1.0')
from gi.repository import Gst
import hailo
from post_process import PoseEstPostProcessing

def run(frame):
    """
    Run Post Processing Steps on Hailo Raw Detections.
    
    Resources:
        https://github.com/hailo-ai/tappas/blob/master/core/hailo/plugins/python/hailo_python_api_sanity.py
    
    Args:
        frame: gsthailo.video_frame.VideoFrame 
            (https://github.com/hailo-ai/tappas/blob/master/core/hailo/python/gsthailo/video_frame.py).

    Returns:
        Gst.FlowReturn.OK    
    """
    
    print('--------CALLBACK STARTED------------------')
    
    post_process = PoseEstPostProcessing(
        max_detections=50,
        score_threshold=0.1,
        nms_iou_thresh=0.65,
        regression_length=15,
        strides=[8, 16, 32]
    )

    # roi = frame.roi
    # buffer = frame.buffer
    # video_info = frame.video_info
    # tensors = roi.get_tensors()
    # objects = roi.get_objects()
    try:
        roi = frame.roi
        raw_tensors = roi.get_tensors() # List[hailo.HailoTensor]
        objects = roi.get_objects()
        
        tensors: Dict[str, np.array] = {}
        for t in raw_tensors:
            # print(tensor.name()) # Returns output layer name
            layer_name = t.name()
            tensor = roi.get_tensor(layer_name)
            tensor_np = np.array(tensor)
            tensor_np = tensor_np.reshape(1, *tensor_np.shape)
            tensors[layer_name] = tensor_np
            print(tensor_np)
        if len(tensors) != 9:
            print("WARNING: tensors does not match with post process step")
            return
        else:
            # result = post_process.post_process(tensors, 640, 640, 1)
            result = post_process.post_process(tensors, 640, 640, 1, 17, normalize=True)
            # print(result['bboxes'])
            # hailo_detections = []
            # for (bbox, score, keypts, joint_score) in zip(result['bboxes'], result['scores'], result['keypoints'], result['joint_scores']):
            #     xmin, ymin, w, h = [float(x) for x in bbox]
            #     bbox = hailo.HailoBBox(xmin, ymin, w, h)
            #     detection = hailo.HailoDetection(bbox, 0, score)
                
            #     hailo_points = [] # List[hailo.HailoPoint]
            #     for pt in keypts:
            #         hailo_points.append(hailo.HailoPoint(pt[0], pt[1], joint_score))
                    
            #     landmarks = hailo.HailoLandmarks('yolo', hailo_points, 0.0)
                    
            #     detection.add_object(landmarks)
            
            # roi.add_objects(hailo_detections)
    except Exception as err:
        print("Error ", err)
        return
    print('+++++++++++++++++++CALLBACK STARTED++++++++++++++++++++++')
    return Gst.FlowReturn.OK

And due to inconsitent outputs post processing is affected and get NaN in boxes.

I have also tried passing output-format-type in hailonet. Because in HailoApplicationCode the output is set to FLOAT32. See here(Hailo-Application-Code-Examples/runtime/python/pose_estimation/pose_estimation.py at a10db4d6799a4355dd71dc7bb1e99887757fc199 · hailo-ai/Hailo-Application-Code-Examples · GitHub)

This is what i tried…

But this also does not work…

[saurabh]

@omria
Reason: The output from the hailonet is quantized value. And this needs to be dequantized using qp_zp and qp_scale params. that we get from vstream_info from tensor(HailoTensor instance).

hailo-ai/Hailo-Application-Code-Examples/blob/a10db4d6799a4355dd71dc7bb1e99887757fc199/runtime/cpp/pose_estimation/yolov8_pose/yolov8pose_postprocess.cpp#L123-L131

This code is alomost used everywhere in c++ implementation. All cpp post processing so libs files are doing the same thing. Either there are this same method or those are utilizing the dequantize functions from “common” header file.
One more example here tappas/core/hailo/libs/postprocesses/depth_estimation/depth_estimation.cpp at daffd36ecab5110d47107255fd7ec4c779758f2e · hailo-ai/tappas · GitHub

But the same thing i can not use. I am unable to get quant_info from vstream in hailopython module.

Can you provide me solution how to deal with those values in the module that i am passing in hailopython.

[victorc]

Hi,
As explained in tappas/docs/write_your_own_application/write-your-own-python-postprocess.rst at master · hailo-ai/tappas · GitHub , calling my_array = np.array(my_tensor, copy=False), should automatically dequantize the tensor from integer format to float.

Note that we have yolov8_pose_estimation post-processing plugin implemented in CPP: tappas/core/hailo/libs/postprocesses/pose_estimation/yolov8pose_postprocess.cpp at master · hailo-ai/tappas · GitHub
You can use it by adding the gstreamer element

hailofilter name=pose-estimation so-path=$landmarks_postprocess_so

to your pipeline after hailonet.

$landmarks_postprocess_so should be equal to $TAPPAS_WORKSPACE/apps/h8/gstreamer/libs/post_processes/libyolov8pose.so

[saurabh]

@victorc
Thank you for the response. I will try adding copy=False.

I have also issue with yolov8_pose_postprocess due to quantization and please see this thread…

Hey I want to build my own custom postprocessing .so

@omria I have figured out the post process issue with yoolv8_pose on my trained model. See the keypoints output type. those are different. [image] To confirm this issue. I build another hef with all 8bit. And this works. In my case I need as much accurate and precise value I can get. So can you tell me at what places I need to modify the code to handle this 16bit data. I applied modifications at a few places but no change. Actual Issue with 16 Bits - In c++ post processing, those keypoi…

Due to this the points are incorrect. And I am unable to use 16bit precision mode with that. Can you tell me how to fix that? I can update and rebuild the so library. I will also prefer to use library if this issue is solved.

[saurabh]

@victorc
The bbox values and confidence are correct. only issue with keypoints positions and confidence values.

[saurabh]

@victorc
I have build .so library with my custom keypoints which is working with 8bit mode but not with 16bit.

[victorc]

You must set the output-format-type argument of hailonet to HAILO_FORMAT_TYPE_UINT16. But the drawback is that it will set all the outputs to UITN16 and in your model, you have only a few outputs set to 16 bits. Could you redo the model conversion to HEF by setting all the outputs to 16-bits ?

[saurabh]

I can try. But I believe this will make things little slower. I am working with 2 cameras and 2 parallel network design.

Do you think I can modify post process code to adapt 16bit for keypoints only? I actually tried few things. but nothing worked.

Thanks again. I will try and update you.

[victorc]

If you set to 16-bits only the very last output node of each output branch, it may not be that slower since there is little or no compute happening.
Since it is not possible to set the Hailo-net bitdepth per output layer, there is no possibility to workaround the issue via the post-process code.

[saurabh]

Thank you I will try changing the precision_mode for all output layers.

FYI, unfortunately setting copy=False did not work.

And the numpy dtype is uint8 still and data also seems siimlar.

[saurabh]

@victorc

I build the model with 16bit precision mode. But it also not working.
Even boxes does not appear now.

I still believe it’s problem is with post process. The way it is handled in the c++ code.

This 16bit working with python processing. But I can not use that approach. Due to speed and higher cpu usage(I have tested).

I am working with 2 cameras and a parallel gstreamer pipeline(with 8 bit it’s working). But 16-bit will be really great for my application.

If you could suggest to me the fix that would be great.

[victorc]

Hi @saurabh ,
If you want to use CPP post-processing instead of Python, you’ll probably need to modify it to support 16-bits tensor.

[saurabh]

@victorc

Thank you. I prefer to use 16-bit in cpp. I can do changes and rebuild. But I am really looking for more info where I need to update the methods, types etc. That would be very helpful.

I tried to check a few other post processing codes, but all were only for 8 bit. I believe. So I was unable to get any help from those.