Hello Hailo Community,
I’m encountering a problem while trying to utilize two Hailo8 chips in parallel using C code with hailort version 4.18.0.
Initially, my code sets up both chips (identified by index 0 and index 1) through the following steps, performed for each chip:
- Scanning for devices and obtaining device IDs:
hailo_device_id_t device_id[MAX_DEVICES]; hailo_scan_devices(NULL, device_ids, &actual_count);
(to get device_ids[0] and device_ids[1] to differentiate the chips.)
- Creating device object:
hailo_device device = NULL; hailo_create_device_by_id(&device_ids[0], &device);
- Creating hef object.
hailo_hef hef = NULL; hailo_create_hef_file(&hef, HEF_FILE); // HEF_FILE is yolov7.hef.
hailo_configure_params_t config_params = {0}; hailo_init_configure_params_by_device(hef, device, &config_params);
hailo_configured_network_group network_group = NULL;
size_t network_group_size = 1; hailo_configure_device(device, hef, &config_params, &network_group, &network_group_size);
hailo_input_vstream_params_by_name_t input_vstream_params[MAX_EDGE_LAYERS] = {0}; hailo_output_vstream_params_by_name_t output_vstream_params[MAX_EDGE_LAYERS] = {0}; hailo_make_input_vstream_params(network_group, false, HAILO_FORMAT_TYPE_AUTO, input_vstream_params, &input_vstreams_size); hailo_make_output_vstream_params(network_group, false, HAILO_FORMAT_TYPE_AUTO, output_vstream_params, &output_vstreams_size);
7.
```c
hailo_input_vstream input_vstreams_temp[MAX_EDGE_LAYERS] = {NULL}; hailo_output_vstream output_vstreams_temp[MAX_EDGE_LAYERS] = {NULL}; hailo_create_input_vstreams(network_group, input_vstream_params, input_vstream_size, input_vstreams_temp); hailo_create_output_vstreams(network_group, output_vstream_params, output_vstream_size, output_vstreams_temp);
hailo_activated_network_group activated_network_group = NULL; hailo_activate_network_group(network_group, NULL, &activated_network_group);
hailo_input_vstream input_vstream_chip0;
hailo_output_vstream output_vstream_chip1;
input_vstream_chip0 = input_vstreams_temp[0]; output_vstream_chip0 = output_vstreams_temp[0];
After setting up the devices and vstreams, I create two pairs of threads using pthread_create() for each chip: a WRITE THREAD that calls hailo_vstream_write_raw_buffer() and a READ THREAD that calls hailo_vstream_read_raw_buffer(). The corresponding input and output vstreams for each chip are passed as arguments when creating threads.
The WRITE THREAD attempts to acquire a semaphore using sem_wait() before calling hailo_vstream_write_raw_buffer(). The READ THREAD, upon returning from hailo_vstream_read_raw_buffer(), signals the semaphore using sem_post(). This semaphore is initialized to 1, allowing the hailo_vstream_write_raw_buffer() to be called initially.
The Problem:
For the first few loops, both chips’ threads operate correctly. However, after a certain number of loops (the exact loop count seems random), the READ THREAD associated with the output vstream of device_ids[0] gets stuck in the hailo_vstream_read_raw_buffer() call and never returns. No timeout errors or any other indications of failure are reported. I can only terminate the process using CTRL+C; otherwise, it remains in this waiting state indefinitely.
Interestingly, the issue only occurs when using both chips in parallel. If I run the same WRITE THREAD and READ THREAD setup for just a single chip, the process runs continuously without any hangs.
Another peculiar observation is that the hang specifically affects the hailo_vstream_read_raw_buffer() call for the output vstream of device_ids[0]. The threads associated with device_ids[1] continue operating normally.
I noticed in the hailort user guide’s changelog for version 4.19 that a bug related to output vstreams in libhailort was mentioned as being fixed. Consequently, I updated hailort to version 4.19 and re-ran my code, but unfortunately, the exact same problem persists.
Could the Hailo technical support team please provide some guidance on how to resolve this issue? Any insights or suggestions for debugging this behavior would be greatly appreciated.
Thank you for your time and assistance.