Bare bones python object detection API

I am a coach with a high school FIRST Robotics Team looking to equip my students with some new tools for the upcoming season. FRC already has various API tools for camera streams and network communication built into their ecosystem that we would like to integrate with the Hailo processor on a Raspberry Pi via the AI kit.

If we already have an existing pipeline that is extracting frames from a camera into an OpenCV mat and has the ability to communicate information (object type, location, etc.) and modified frames (image + bounding boxes and other annotations), is there a simple way to use the Python Hailo API to run object detection on the frame and access the resulting inference results without requiring all of the other framework used in the examples?

Yes, exactly as you’ve written, you can use the direct Python API, would be something like this example:
Hailo-Application-Code-Examples/runtime/python/object_detection/object_detection.py at main · hailo-ai/Hailo-Application-Code-Examples

Thanks for the quick reply. I assume most of the relevant code in this example is in the infer function. Can this be simplified further if we are trying to just process a single frame at a time synchronously without needing to build up a queue and starting up a separate thread?

I was able to reverse engineer a lot of the code in HailoAsyncInference in utils.py from the example github, and can now create the key objects like the VDevice, HEF, and ConfiguredInferModel. I see ConfiguredInferModel has a synchronous run() method, so it seems the last thing I’m missing is the format/contents of the bindings object.

Can anyone explain in more detail the structure of the input/output buffers or provide a simple example of a call to ConfiguredInferModel.create_bindings which would work in the use case outlined in this thread?

Thanks in advance!

I second @billjester3 request. While object detection example for Python is great it is also not so great for entry point users. A wast amount of users prefer to do synchronous processing (me included) because they want to show the result of inference in real time on the video.

Can we get a simplified example please?

Another question: where is HailoRT documentation? I am not talking about the user guide, but programmers guide - you know, every function with parameters described. A documentation that can be browsed and searched.

Thank you.

I whish I could edit my posts… :frowning:

There is some API documentation at HailoRT v4.19.0, but it’s not super detailed.

@billjester3
I’ve managed to do this, but this is just the second day of my experience with Hailo so I might be imprecise / sub-optimal in some things. I am assuming you read the docs and you know what are output_buffers and input_buffers.

The minimal example depends on the type of model that you will use, some models require more pre-processing than others.

Basically, what you need to do is to take the object_detection.pyexample and extract from it _create_bindings. This will create you bindings with output_buffers. It will detect the output layer of your AI model.

The next thing you need is utils.preprocess() method to prepare image for consumption by your model :wink:

After you preprocess the image, you need to pass it to the input buffer. After passing it, you can execute run() method.

Then you need to call buffer = bindings.output().get_buffer() to get detections. Then you need to post process them to get bounding boxes.

Hope this help.

That’s great, thanks for the update. I started extracting pieces from utils.java as well and was able to get it to run without error, but I never got an actual output result. I’m guessing the image going to the input buffer is formatted incorrectly. I was trying it with the yolov8s_h8l.hef model.

Do you have any sample code you can share, even if it’s just a first pass?

@Chuck_Rhoades , do you have any sample code you can share? I had some success using the Picamera2 integration at picamera2/examples/hailo/detect.py at main · raspberrypi/picamera2 · GitHub, but this seems like it’s still a work in progress and the inference results are unstable. I would like to try again with the Hailo API but still need to figure out the right formatting of the input and output buffers/bindings passed to ConfiguredInferModel.run().

Thanks!

@billjester3
Not sure if you are still looking for simple APIs to run inference, but we developed a python SDK for this purpose. Please see Simplifying Edge AI Development with DeGirum PySDK and Hailo