Best approach for multi-model video inference in rpi

I’ve been playing around with the Hailo8 on the rpi for a couple of weeks now and I’m struggling with performance issues when I try modifying some of the applications in the Application code examples. Specifically, I took the detection_with_tracker example and pulled in the inference and postprocessing from the pose_estimation and removed tracking.

When I ran it without the postprocessing I could only get 28 FPS which is a bit disappointing as when I run the model using:

hailortcli run yolov8s_pose.hef

It says I should be able to get 56 FPS.

I then added the postprocessing (copied from pose_estimation_utils.py) and the FPS dropped to 12. This seemed pretty crazy.

I have tried spinning up a separate process for the postprocessing but then the hailo device has a hissy fit.

I then went down an extremely frustrating several hour rabbit hole of trying to get the model zoo installed on the PI which was fraught with problems. I was going to try to use the built in postprocessing but looking at the code I doubt it will be much more efficient.

I have tried running pose detection in 2 other ways: using the gstreamer example and using a 3rd party library degirum and in both cases I could get higher frame rates even with post processing.

In the case of degirum I even managed to easily get 3 separate models running concurrently (pose, object and face) and it ran at about 24 FPS.

I’m trying not do rely on degirum as I want to be able to fully understand what is happening and be a bit closer to the metal.

For the PI is it recommended to use gstreamer? I would assume so as all of the examples given are done with that. I’m kind of hoping I can avoid this as I already have enough new stuff to figure out without having to learn gstreamer.

My final application will need to have pose, object and face detection running inference on a camera so any advice on this will be greatly appreciated.

Hi @rosslote
Glad to see you got things working using DeGirum PySDK. Our PySDK is a wrapper over the HailoRT to make development easier. Based on our experience with multiple edge hardware options, we designed PySDK to make it easy for users to develop applications without sacrificing performance. If you can let us know what you want to understand about PySDK to feel comfortable, we will try our best to explain.