Handling Input and Output for Asynchronous Inference with Hailort

Hyunha_Hwang · April 17, 2025, 3:24am

I am currently working on implementing asynchronous inference using Hailort, following the examples provided in the links below:

github.com/hailo-ai/hailort

hailort/libhailort/examples/cpp/async_infer_basic_example/async_infer_basic_example.cpp

master

/**
 * Copyright (c) 2019-2025 Hailo Technologies Ltd. All rights reserved.
 * Distributed under the MIT license (https://opensource.org/licenses/MIT)
 **/
/**
 * @file async_infer_basic_example.cpp
 * This example demonstrates the Async Infer API usage with a specific model.
 **/

#include "hailo/hailort.hpp"

#include <iostream>

#if defined(__unix__)
#include <sys/mman.h>
#endif

#define HEF_FILE ("hefs/shortcut_net.hef")

using namespace hailort;

This file has been truncated. show original

github.com/hailo-ai/hailort

hailort/libhailort/examples/cpp/async_infer_advanced_example/async_infer_advanced_example.cpp

master

/**
 * Copyright (c) 2019-2025 Hailo Technologies Ltd. All rights reserved.
 * Distributed under the MIT license (https://opensource.org/licenses/MIT)
 **/
/**
 * @file async_infer_advanced_example.cpp
 * This example demonstrates the Async Infer API usage with a specific model that has multi-planar input
 * and changes configutrations of the streams.
 * Multiple infer jobs are triggered, and waiting for the last one ensures that all the rest will arrive as well.
 **/

#include "hailo/hailort.hpp"

#include <iostream>

#if defined(__unix__)
#include <sys/mman.h>
#endif

#define BATCH_COUNT (100)

This file has been truncated. show original

However, I have encountered an issue as I am unable to find any examples on how to write inputs (such as images) and read the corresponding outputs during the asynchronous inference process.

Could you please provide me with a working example or guidance on how to handle image input and output properly in an asynchronous inference pipeline? It would be especially helpful if you could show how to maximize the NPU usage for optimal performance during this process.

Any additional resources or advice on input/output handling for NPU-accelerated inference would be greatly appreciated.

Thank you for your time and support.

omria · April 20, 2025, 4:07pm

Hey @Hyunha_Hwang ,

Welcome to the Hailo Community!

Please check out our example in here : Hailo-Application-Code-Examples/runtime/cpp/object_detection at main · hailo-ai/Hailo-Application-Code-Examples · GitHub

This has the input out and all of the api handling .

Topic		Replies	Views
Working python hailo inference General	15	751	March 18, 2025
Multithreading inference of two models on a single Hailo-8 General hailort , hailo8	6	220	January 2, 2025
Working Python inference example with .hef on Raspberry Pi (HailoRT 4.21.0, critical request) General hailort , raspberry-pi , hailo8	6	226	May 22, 2025
HailoRT minimal working example for Python and Hailo8 Guides hailort , raspberry-pi , hailo8	5	2386	May 14, 2025
DFC_4_Inference_Tutorial error General hailort , raspberry-pi , hailo8 , error	1	49	January 29, 2025

Handling Input and Output for Asynchronous Inference with Hailort

Related topics