Help with using models from the Model Zoo

General
1

Hi everyone,

I realize this might be a bit of a basic question, and I’m sorry in advance if it comes off as annoying or if it’s already been answered somewhere.

I’m completely new to AI accelerators and the whole topic of running neural networks. I recently started working with the Hailo platform but I’m a bit stuck on how to actually run one of the models from the Hailo model zoo using Python.

I was able to run the tests from the hailo-rpi5-examples but I want to use the damoyolo and the mspn_regnetx_800mf models. I found the precompiled .hef files but I found nothing about actually running the model and post processing the results.

tl;dr:
My Questions:

How do I load and run a precompiled .hef model (for example damoyolo or mspn_regnetx_800mf) from the model zoo in a Python script?

What’s the typical workflow to feed data into the model and get results back?

And most importantly: how do I interpret the model outputs (e.g., do post-processing for object detection or classification)?

I’ve been digging through the docs and examples, but as a beginner it’s still a bit overwhelming. If someone could point me to a simple example or explain the rough steps, that would help me out a lot.

Thanks in advance,
BatiFPV

2

Hey @BatiFPV,

Welcome to the community!

First thing you’ll need to do is add your model to the simple detection pipeline run. Then you’ll have to create a custom post-process for your specific hef , check out this guide ,its really helpful for that: hailo-apps-infra/doc/developer_guide/writing_postprocess.md at main · hailo-ai/hailo-apps-infra · GitHub

Once you’ve got your post-process written, you’ll need to add it to the cpp_postprocess directory: hailo-apps-infra/hailo_apps/hailo_app_python/core/cpp_postprocess at main · hailo-ai/hailo-apps-infra · GitHub

Don’t forget to make sure you have the json file with your labels set up properly - that tripped me up the first time!

Let me know if you run into any issues with any of these steps, happy to help troubleshoot.

1 Like
3

Hi @omria!
Thanks for trying to help me! :slight_smile:
I finally got to the point where I was able to preprocess an image, run the model, and get back the results.
But i couldn’t find anything about how to postprocess the results to get anything useful.
The only thing I was able to find is this:

but for some reason even after a long time of debugging i get nothing useful :frowning:
Where can I find anything about what the model is returning and how I need to postprocess it?
The model returns 6 dicts which are presumably 3 pairs of task projection layers, one cls and one reg. But I have no idea how to work with the data.
Furthermore, I read that I need to dequantize the values before I can work with them, but where can I find the scale and zero point values for doing that? Are they embedded in the .hef file and I can read them back somehow?
I use the precompiled damoyolo_tinynasL25_S.hef file from the hailo model zoo.
Thanks again!
BatiFPV

4

Hi @BatiFPV
I am from the DeGirum team. We will help you with this model. While our team prepares an example that uses built-in postporcessor for DamoYOLO, you can take a look at how to use this model with our pysdk here: hailo_examples/examples/damo_yolo.ipynb at main · DeGirum/hailo_examples

2 Likes
5

Wow, that was a quick reply thank you!
I already looked at the example in your pysdk but I was wondering if would be possible to run the post processing on top of the HailoRT output tensor, because I already got the right output (at least i think so :D) and I only need to do the post processing.

6

Hi @BatiFPV
yes, you can run he postprocessor on top of the HailoRT output tensor. In fact, that is what our PySDK does and all the code is shared. Since, it is hard for us to debug and support each individual use case, we provide out-of-the-box running examples using our PySDK. If for any reason, you do not want to use PySDK, you can still use our postprocessor code but we cannot help you debug due to limited resources on our side.

7

Oh I absolutely understand that! Thank you anyways!
Can you tell me, where and how the data is going into your post processor? So is the part before the post processor on github as well? I am pretty sure that would help me alot!

8

Hi @BatiFPV
The main function that performs the postprocessing is the forward function of the PostProcessor class. It takes a list of tensors (output tensors from HailoRT) named as tensor_list, and a list of quantization parameters named as details_list. You can copy+paste the code from our postprocessor and modify it so that you can use the functions directly. Hope this helps.

9

Is there a way to get the quantization parameters from a compiled .hef file?
The output tensors that I get from HailoRT is in the form of a dict:
dict_keys([‘damoyolo_tinynasL25_S/conv59’, ‘damoyolo_tinynasL25_S/conv60’, ‘damoyolo_tinynasL25_S/conv71’, ‘damoyolo_tinynasL25_S/conv72’, ‘damoyolo_tinynasL25_S/conv82’, ‘damoyolo_tinynasL25_S/conv83’])
(1, 80, 80, 68)
(1, 80, 80, 81)
(1, 40, 40, 68)
(1, 40, 40, 81)
(1, 20, 20, 68)
(1, 20, 20, 81)
Do i need to reformat that into a list or is .values() enough?
Thank you again, you already helped alot!

10

Hi @BatiFPV
Below is code snippet to get the quantization parameters:

from hailo_platform import HEF

hef = HEF("your_hef_path")
output_vstream_info = hef.get_output_vstream_infos()

print("Outputs")
for output_info in output_vstream_info:
  print(output_info)
  print("Scale: {}".format(output_info.quant_info.qp_scale))
  print("Zero point: {}\n".format(output_info.quant_info.qp_zp))

You can change the postprocessor code to accept a dict instead of a list but in any case the conversion should be easy.

11

Thank you very much! I finally managed to get it working properly!
I noticed that it isn’t possible to run the post processing in realtime on a rpi5 without dropping frames. The cpu is pinned to 100%, is that normal / is it possible to optimize it?

1 Like
12

Hi @BatiFPV
Yes, postprocessing is CPU intensive and if you do it in python it is not only slow but also consumes CPU resources. You can still try our PySDK which has a built-in c++ postprocessor for damoyolo (no pressure of course :slight_smile: ). While this also runs on CPU, it is much faster and does not max out CPU.