Handling Input and Output for Asynchronous Inference with Hailort

Hyunha_Hwang · April 17, 2025, 3:24am

I am currently working on implementing asynchronous inference using Hailort, following the examples provided in the links below:

github.com/hailo-ai/hailort

hailort/libhailort/examples/cpp/async_infer_basic_example/async_infer_basic_example.cpp

master

/**
 * Copyright (c) 2019-2025 Hailo Technologies Ltd. All rights reserved.
 * Distributed under the MIT license (https://opensource.org/licenses/MIT)
 **/
/**
 * @file async_infer_basic_example.cpp
 * This example demonstrates the Async Infer API usage with a specific model.
 **/

#include "hailo/hailort.hpp"

#include <iostream>

#if defined(__unix__)
#include <sys/mman.h>
#endif

#define HEF_FILE ("hefs/shortcut_net.hef")

using namespace hailort;

This file has been truncated. show original

github.com/hailo-ai/hailort

hailort/libhailort/examples/cpp/async_infer_advanced_example/async_infer_advanced_example.cpp

master

/**
 * Copyright (c) 2019-2025 Hailo Technologies Ltd. All rights reserved.
 * Distributed under the MIT license (https://opensource.org/licenses/MIT)
 **/
/**
 * @file async_infer_advanced_example.cpp
 * This example demonstrates the Async Infer API usage with a specific model that has multi-planar input
 * and changes configutrations of the streams.
 * Multiple infer jobs are triggered, and waiting for the last one ensures that all the rest will arrive as well.
 **/

#include "hailo/hailort.hpp"

#include <iostream>

#if defined(__unix__)
#include <sys/mman.h>
#endif

#define BATCH_COUNT (100)

This file has been truncated. show original

However, I have encountered an issue as I am unable to find any examples on how to write inputs (such as images) and read the corresponding outputs during the asynchronous inference process.

Could you please provide me with a working example or guidance on how to handle image input and output properly in an asynchronous inference pipeline? It would be especially helpful if you could show how to maximize the NPU usage for optimal performance during this process.

Any additional resources or advice on input/output handling for NPU-accelerated inference would be greatly appreciated.

Thank you for your time and support.

omria · April 20, 2025, 4:07pm

Hey @Hyunha_Hwang ,

Welcome to the Hailo Community!

Please check out our example in here : Hailo-Application-Code-Examples/runtime/cpp/object_detection at main · hailo-ai/Hailo-Application-Code-Examples · GitHub

This has the input out and all of the api handling .

Topic		Replies	Views
HailoRT minimal working example for Python and Hailo8 Guides hailort , raspberry-pi , hailo8	5	2538	May 14, 2025
Can we get a "Hello World" tutorial for writing our own AI app? General	3	238	September 29, 2024
Working python hailo inference General	15	782	March 18, 2025
Hailo8L Async Inference issue - invalid operation ( General hailort	1	139	March 24, 2025
Issue with running multiple output model General hailort , hailo8	4	275	August 11, 2024

Handling Input and Output for Asynchronous Inference with Hailort

Related topics