hailo-8 Gstreamer integration in python with specialized model

General
, ,
1

Hi Hailo community,

I have official AI Hat for Raspberry Pi with the Hailo-8, and I’ve successfully compiled/converted my model from .onnx to .hef. The model I’m using for testing is version-RFB-640.onnx, which I found in a GitHub repository. This model is intended for face recognition, but just to clarify, I don’t plan to use it in my final app—it’s just for testing purposes. I want to compile it to .hef, run it, and verify the output in Python so that I can train before developing my actual application.

When I ran the hailo parse-hef command, I received the following output:

(hailo) Running command ‘parse-hef’ with ‘hailortcli’
Architecture HEF was compiled for: HAILO8
Network group name: version-RFB-640, Single Context
Network name: version-RFB-640/version-RFB-640
VStream infos:
Input version-RFB-640/input_layer1 UINT8, NHWC(480x640x3)
Output version-RFB-640/conv21 UINT8, FCR(60x80x12)
Output version-RFB-640/conv34 UINT8, FCR(8x10x12)
Output version-RFB-640/conv33 UINT8, NHWC(8x10x6)
Output version-RFB-640/conv26 UINT8, NHWC(30x40x8)
Output version-RFB-640/conv30 UINT8, FCR(15x20x4)
Output version-RFB-640/conv20 UINT8, NHWC(60x80x6)
Output version-RFB-640/conv25 UINT8, NHWC(30x40x4)
Output version-RFB-640/conv31 UINT8, FCR(15x20x8)

Now, I want to run this model from Python using GStreamer. I found the apps-infra/gstreamer_helper_pipelines.py file, which helped me create my first pipeline. However, I ran into some issues when trying to execute it. To simplify, I ended up with this pipeline:

pipeline_str = “”"
libcamerasrc ! videoconvert ! videoscale ! video/x-raw,format=RGB,width=640,height=480 !
hailonet hef-path=/version-RFB-640.hef batch-size=20 device-count=1 is_active=true ! appsink name=debug_sink
“”"

This works without errors, but it only outputs an image. According to the HailoNet docs, the model output should be in the form of metadata, but I couldn’t find more details or examples on how to extract this metadata. So, I’m unsure whether it’s working as expected.

Initially, I had autovideosink, but I got the following error:
HailoNet Error: gst_pad_push failed with status = -4.

I’ve also seen that hailofilter might help extract metadata from the pipeline, but it requires a so-path argument, and I have no idea what this .so file is or how to create one. I heard you can do this using Cython, but I’m lost when it comes to writing the .so file for my model.

At this point, I’m a bit confused and would really appreciate any help or guidance to move forward. If anyone has worked with similar pipelines or knows how to handle the metadata output, it would be a huge help!

Thanks in advance! :blush:

2

This is a late reply. The issue is that hailonet Gstreamer plugin adds model as metadata to the buffer that is past to downstream gstreamer elements. This data is “invisible” to regular gstreamer elements .

You are on the correct path to look at hailofilter. This plugin knows how to extract the model output from metadata. The .so your refer to is Shared Object library that you compile from your own C code that will be called by hailofilter for every frame that is processed. It receives a HailoROI object which allows you to extract the tensor(s) that were output by the model in hailonet. There is a tutorial to write this code, and build it at tappas/docs/write_your_own_application/write-your-own-postprocess.rst at master · hailo-ai/tappas · GitHub

If you would rather work in python, tappas also supplies a ‘hailopython’ gstreamer plugin. There you specify the python module and function to be called for every frame received by hailopython. Your python function also gets a ROI object from which you can get the model output tensor(s) convert them to numpy arrays. The tutorial for this approach is tappas/docs/write_your_own_application/write-your-own-python-postprocess.rst at master · hailo-ai/tappas · GitHub