Hi,
I’m running a 24/7 object detector for IP cameras on a Raspberry Pi 5 with the AI HAT+ 2 (Hailo-10H). Every few hours, inference stops in all processes at the same moment. Nothing is logged by HailoRT or the kernel, and nothing recovers on its own. Restarting the processes fixes it until the next time.
Setup
- Raspberry Pi 5 Model B Rev 1.1, Raspberry Pi OS Trixie (Debian 13), kernel
6.18.50+rpt-rpi-2712(16K pages) - h10-hailort / h10-hailort-pcie-driver / python3-h10-hailort 5.1.1, hailo-tappas-core 5.1.0, hailo-apps detection pipeline
- Model:
yolov8m.hef(hailo10h) with the hailo-apps detection pipeline:hailonet+hailofilter+hailotracker - 3 separate processes, one per RTSP camera, each with its own vdevice:
scheduling_algorithm: ROUND_ROBIN, multi_process_service: false. Each runs at 10 fps (videorate), with 640×480 or 1280×720 input scaled to the model. - Temperature ~63 °C,
vcgencmd get_throttled= 0x0, ~15 GB RAM free, load ~0.5–0.8
Symptom
- About 2–3 times a day, at irregular intervals of roughly 5–11 h: Oct 31 13:37, 16:18; Nov 1 07:03, 10:43, 19:47; Nov 2 01:34, 06:14, 11:32, 20:32; Nov 3 07:12, 12:35, 19:34.
- All three processes stop getting inference results within the same second.
- The cameras and the network are fine during the freeze. Continuous ping logging shows normal latency, and separate ffmpeg recorders on the same cameras keep receiving.
- My own watchdog restarts the processes after a timeout, and inference works again right away.
What I captured at the moment of the freeze (pad probes on every queue’s src pad plus hailonet, /proc/self/task/*/wchan, a 2 s CPU sample). It was identical in all 12 reports (4 freezes × 3 processes):
== buffer flow (oldest output first)
inference_hailonet last out 63.0s ago buffers 250230
inference_hailofilter_q last out 63.0s ago buffers 250230 level 0/3
... (everything downstream: same, empty)
inference_wrapper_bypass_q last out 63.0s ago buffers 250231 level 20/20
inference_hailonet_q last out 61.1s ago buffers 250251 level 0/3
inference_wrapper_input_q last out 60.9s ago buffers 250252 level 3/3
source_convert_q last out 60.5s ago level 3/3
source_queue_decode last out 60.0s ago level 3/3
== threads (2 s sample)
1 x Hailo Detection S hailo_vdma_context_wait_for_interrupt cpu +0
(all others idle on futex/poll, cpu +0)
== kernel log (last 5 min): empty
So hailonet accepted about 20 more frames after its last output, then stopped returning results, and everything upstream filled up through back-pressure. One thread waits in hailo_vdma_context_wait_for_interrupt and never returns. CPU is 0 everywhere. HailoRT logs nothing, dmesg shows nothing, and the pipeline still reports PLAYING.
Questions
- Is this a known issue with HailoRT / the Hailo-10H driver 5.1.1, e.g. a missed vDMA interrupt or a scheduler deadlock when several processes share the device?
- Is running several separate processes, each with its own vdevice and no multi-process service, a supported setup for the Hailo-10H? Would one process with a multi-source pipeline be more reliable?
- Is there a newer HailoRT / driver / firmware for the Hailo-10H I should try, or any driver parameter or log level that would show more at the moment of the hang?
I’m currently testing with only two processes to see whether the frequency changes, and I’ll report back. Full freeze reports are available.
Thanks!