I’m building a player application around GStreamer on Windows where each video stream runs in its own process and performs inference with the same model on a shared Hailo-8. To let multiple player processes share the device, I want to use HailoRT’s scheduler through the multi-process service: every hailonet element configured with multi-process-service=true, the same vdevice-group-id, and scheduling-algorithm=1 (ROUND_ROBIN).
When running, a single hailonet element with multi-process-service=true fails in the first frame, even with no second process competing for the device. The vdevice creation and firmware identify both succeed over the gRPC channel to hailort_service.exe, but the first ConfiguredNetworkGroup_infer_async call errors out with HAILO_INTERNAL_FAILURE(8) and the async pipeline shuts down.
The same pipeline with multi-process-service=false runs perfectly. hailortcli run --multi-process-service runs perfectly at ~300 FPS through the same service, with the same model. So hailoRT seems to work fine, it’s specifically the hailonet service path that seems to break.
Windows 11 Pro (10.0.26200.x)
M.2 Hailo-8 module
GStreamer 1.26.2 MinGW x64 (installed via MSVC installer)
HailoRT 4.23.0, with all features enabled
MWE:
gst-launch-1.0 -v ^
videotestsrc num-buffers=120 is-live=false pattern=ball ^
! video/x-raw,format=RGB,width=640,height=640,framerate=30/1 ^
! queue leaky=downstream max-size-buffers=2 ^
! hailonet hef-path=“C:/path/to/yolov8n.hef” ^
batch-size=1 ^
force-writable=true ^
device-count=1 ^
vdevice-group-id=SHARED ^
scheduling-algorithm=1 ^
scheduler-priority=16 ^
scheduler-threshold=1 ^
scheduler-timeout-ms=100 ^
multi-process-service=true ^
! fakesink sync=false
Stderr I get:
Setting pipeline to PAUSED …
Pipeline is PREROLLING …
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
Redistribute latency…
New clock: GstSystemClock
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INTERNAL_FAILURE(8) -
[HailoRT] [error] Non-recoverable Async Infer Pipeline error. status error code: HAILO_INTERNAL_FAILURE(8)
[HailoRT] [error] Shutting down the pipeline with status HAILO_INTERNAL_FAILURE(8)
CHECK_EXPECTED failed with status=8
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
streaming stopped, reason error (-5)
Execution ended after 0:00:00.001697800
hailort.log shows the failure is on the first infer_async over RPC, after vdevice creation and firmware identify have both already succeeded over the same channel:
[hailort::VDevice::create] Creating vdevice with params: device_count: 1, scheduling_algorithm: ROUND_ROBIN, multi_process_service: true
[hailort::control__parse_identify_results] firmware_version is: 4.23.0
[hailort_rpc_client.cpp:1525] [hailort::HailoRtRpcClient::ConfiguredNetworkGroup_infer_async] CHECK_SUCCESS failed with status=HAILO_INTERNAL_FAILURE(8)
[network_group_client.cpp:692] [hailort::ConfiguredNetworkGroupClient::infer_async] CHECK_SUCCESS failed with status=HAILO_INTERNAL_FAILURE(8)
[pipeline_internal.cpp:26] [hailort::PipelineElementInternal::handle_non_recoverable_async_error] Non-recoverable Async Infer Pipeline error. status error code: HAILO_INTERNAL_FAILURE(8)
[async_infer_runner.cpp:88] [hailort::AsyncPipeline::shutdown] Shutting down the pipeline with status HAILO_INTERNAL_FAILURE(8)
What I tried already:
- Same pipeline with multi-process-service=false → runs cleanly to EOS at video-test-source rates. The only delta is the flag.
- hailortcli run --multi-process-service yolov8n.hef → ~300 FPS, zero errors. The service handles inference fine for hailortcli, so the service is working.
- Verified hailort_service is Running (Automatic startup).
- Tried scheduler tuning (scheduler-priority=16, scheduler-threshold=1, scheduler-timeout-ms=100) with no effect. The failure happens on the first inference call, before any scheduler decision is made.
- Tried with two parallel processes (the original use case) and both fail symmetrically with the same HAILO_INTERNAL_FAILURE(8), confirming this is not a sharing-collision: even one process can’t get past the first infer_async over RPC.
- Reinstalled HailoRT 4.23.0 cleanly with all features enabled. Also reinstalled GStreamer, though didn’t expect to do much difference.