Hailo-10H (AI HAT+ 2): hailonet stops returning results in all processes at once every few hours, thread stuck in hailo_vdma_context_wait_for_interrupt

Hi,

I’m running a 24/7 object detector for IP cameras on a Raspberry Pi 5 with the AI HAT+ 2 (Hailo-10H). Every few hours, inference stops in all processes at the same moment. Nothing is logged by HailoRT or the kernel, and nothing recovers on its own. Restarting the processes fixes it until the next time.

Setup

  • Raspberry Pi 5 Model B Rev 1.1, Raspberry Pi OS Trixie (Debian 13), kernel 6.18.50+rpt-rpi-2712 (16K pages)
  • h10-hailort / h10-hailort-pcie-driver / python3-h10-hailort 5.1.1, hailo-tappas-core 5.1.0, hailo-apps detection pipeline
  • Model: yolov8m.hef (hailo10h) with the hailo-apps detection pipeline: hailonet + hailofilter + hailotracker
  • 3 separate processes, one per RTSP camera, each with its own vdevice: scheduling_algorithm: ROUND_ROBIN, multi_process_service: false. Each runs at 10 fps (videorate), with 640×480 or 1280×720 input scaled to the model.
  • Temperature ~63 °C, vcgencmd get_throttled = 0x0, ~15 GB RAM free, load ~0.5–0.8

Symptom

  • About 2–3 times a day, at irregular intervals of roughly 5–11 h: Oct 31 13:37, 16:18; Nov 1 07:03, 10:43, 19:47; Nov 2 01:34, 06:14, 11:32, 20:32; Nov 3 07:12, 12:35, 19:34.
  • All three processes stop getting inference results within the same second.
  • The cameras and the network are fine during the freeze. Continuous ping logging shows normal latency, and separate ffmpeg recorders on the same cameras keep receiving.
  • My own watchdog restarts the processes after a timeout, and inference works again right away.

What I captured at the moment of the freeze (pad probes on every queue’s src pad plus hailonet, /proc/self/task/*/wchan, a 2 s CPU sample). It was identical in all 12 reports (4 freezes × 3 processes):

== buffer flow (oldest output first)
  inference_hailonet            last out  63.0s ago  buffers 250230
  inference_hailofilter_q       last out  63.0s ago  buffers 250230  level 0/3
  ... (everything downstream: same, empty)
  inference_wrapper_bypass_q    last out  63.0s ago  buffers 250231  level 20/20
  inference_hailonet_q          last out  61.1s ago  buffers 250251  level 0/3
  inference_wrapper_input_q     last out  60.9s ago  buffers 250252  level 3/3
  source_convert_q              last out  60.5s ago  level 3/3
  source_queue_decode           last out  60.0s ago  level 3/3

== threads (2 s sample)
    1 x Hailo Detection    S hailo_vdma_context_wait_for_interrupt    cpu +0
    (all others idle on futex/poll, cpu +0)

== kernel log (last 5 min): empty

So hailonet accepted about 20 more frames after its last output, then stopped returning results, and everything upstream filled up through back-pressure. One thread waits in hailo_vdma_context_wait_for_interrupt and never returns. CPU is 0 everywhere. HailoRT logs nothing, dmesg shows nothing, and the pipeline still reports PLAYING.

Questions

  1. Is this a known issue with HailoRT / the Hailo-10H driver 5.1.1, e.g. a missed vDMA interrupt or a scheduler deadlock when several processes share the device?
  2. Is running several separate processes, each with its own vdevice and no multi-process service, a supported setup for the Hailo-10H? Would one process with a multi-source pipeline be more reliable?
  3. Is there a newer HailoRT / driver / firmware for the Hailo-10H I should try, or any driver parameter or log level that would show more at the moment of the hang?

I’m currently testing with only two processes to see whether the frequency changes, and I’ll report back. Full freeze reports are available.

Thanks!

Update: with only two processes (one camera disabled) since Nov 3 19:58, I’ve had 2 freezes in about 48 h (Nov 5 11:53 and 20:12), compared with about 3.7 a day with three processes. The signature is the same: hailonet output stops first, upstream queues fill through back-pressure, one thread sits in hailo_vdma_context_wait_for_interrupt, and both processes stop within about a second. Ping and the separate ffmpeg recorders stay normal. The sample is small, but it looks like fewer processes means fewer freezes, not none. I can provide full reports or run with HailoRT debug logging if that helps

@Robb_Tschi I recommend that you upgrade to the latest Hailo-RT v.5.4.0 on the RPI5 and see if the issue still persists.

To upgrade a Raspberry PI installation to the latest version of Hailo-RT follow these steps:

  1. Hailo Developer Zone (https://hailo.ai/developer-zone/software-downloads/?product=ai_accelerators&device=hailo_10h): Download the latest PCIe driver and HailoRT (filter: AI Software Suite, HailoRT, ARM64, Linux, relevant Python version (e.g. 3.13))
    2 deb files: hailort_5.4.0_arm64.deb , hailort-pcie-driver_5.4.0_all.deb
    1 whl file: hailort-5.4.0-cp313-cp313-linux_aarch64..whl

  2. Hailo Developer Zone: Download the latest Model Zoo (filter: AI Software Suite, Model Zoo, ARM64, Linux, ignore Python version) - 1 deb file: hailo_gen_ai_model_zoo_5.4.0_arm64.deb

  3. TAPPAS (installed with hailo-h10-all by RPi): Hailo Developer Zone: Download the latest TAPPAS version (filter: AI Software Suite, Example Applications, ARM64, Linux, ignore Python version). Download only the deb file: hailo-tappas-core_5.4.0_arm64.deb

  4. Copy all files to a folder ~/Downloads/rpi5-h10h

  5. Open a shell and execute:
    cd ~/Downloads/rpi5-h10h

  6. Remove older hailo releases:
    # Removes also hailo-10h-all
    sudo apt remove h10-hailort-pcie-driver
    sudo apt remove hailo-gen-ai-model-zoo
    # Removes also python3-h10-hailort rpicam-apps-hailo-postprocess
    sudo apt remove h10-hailort

  7. Install PCIe driver, HailoRT, Model Zoo & TAPPAS:
    sudo apt install \
    ./hailort_5.4.0_arm64.deb \
    ./hailort-pcie-driver_5.4.0_all.deb \
    ./hailo_gen_ai_model_zoo_5.4.0_arm64.deb \
    ./hailo-tappas-core_5.4.0_arm64.deb

  8. Restart: sudo reboot

  9. Verify Installation: hailortcli fw-control identify

Install the latest hailo-apps v26.10.0, which is compatible with Hailo-RT version 5.4.0 .

When this document was written, Hailo-RT version 5.4.0 was the latest. By the time you read this, a newer version may be available.

Since your application involves multiple cameras, please use the example “multisource”: hailo-apps/hailo_apps/python/pipeline_apps/multisource at main · hailo-ai/hailo-apps · GitHub

Update 2: Thanks a lot, Victor! After upgrading to HailoRT/driver/firmware 5.4.0, TAPPAS 5.4.0 and hailo-apps 26.10.0, the freezes are gone: three camera processes on the Hailo-10H ran ~71 h without a single hang, compared with ~3.7 per day on 5.1.1. (Note for others: hailo-apps 26.10.0 changed the hailo10h detection default from yolov8m to yolov5m_wo_spp, so pass --hef-path yolov8m if you relied on the old default, and re-download the HEF compiled for 5.4.0.)