Unfortunately, I’m having similar issues as reported in Problems running the hailo raspi5 examples My issue may be a bit different in that I only get a hang without the get_user_pages error logged until I ctl-c out of the demo, then I see the get_user_pages error the second time I try to run a demo model. If I let it run, it just hangs for a couple of minutes when I will get:
May 21 14:40:43 raspberrypi kernel: INFO: task inference_hailo:3223 blocked for more than 120 seconds.
May 21 14:40:43 raspberrypi kernel: Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
May 21 14:40:43 raspberrypi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 21 14:40:43 raspberrypi kernel: task:inference_hailo state:D stack:0 pid:3223 tgid:3192 ppid:3037 flags:0x00000004
May 21 14:40:43 raspberrypi kernel: Call trace:
May 21 14:40:43 raspberrypi kernel: __switch_to+0xf0/0x150
May 21 14:40:43 raspberrypi kernel: __schedule+0x38c/0xb50
May 21 14:40:43 raspberrypi kernel: schedule+0x3c/0x148
May 21 14:40:43 raspberrypi kernel: schedule_preempt_disabled+0x2c/0x50
May 21 14:40:43 raspberrypi kernel: rwsem_down_read_slowpath+0x214/0x500
May 21 14:40:43 raspberrypi kernel: down_read+0xac/0xc0
May 21 14:40:43 raspberrypi kernel: hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
May 21 14:40:43 raspberrypi kernel: hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
May 21 14:40:43 raspberrypi kernel: hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
May 21 14:40:43 raspberrypi kernel: hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
May 21 14:40:43 raspberrypi kernel: __arm64_sys_ioctl+0xb4/0x100
May 21 14:40:43 raspberrypi kernel: invoke_syscall+0x50/0x120
May 21 14:40:43 raspberrypi kernel: el0_svc_common.constprop.0+0x48/0xf0
May 21 14:40:43 raspberrypi kernel: do_el0_svc+0x24/0x38
May 21 14:40:43 raspberrypi kernel: el0_svc+0x30/0xd0
May 21 14:40:43 raspberrypi kernel: el0t_64_sync_handler+0x100/0x130
May 21 14:40:43 raspberrypi kernel: el0t_64_sync+0x190/0x198
I’ve mainly been trying to work with basic_pipelines/detection_simple.py and basic_pipelines/detection.py which work once and then will hang until I restart. (I’m pretty sure detection_simple.py was running more than once but now it seems to only be running once) However, after getting my single successful run of a repo demo model I can switch back to the rpi demos (i.e. rpicam-hello with hailo_yolov8_inference.json) which appear to still be working (except the rpi segmentation demo model apparently isn’t in the package with the other models?) without a reboot as it appears that it’s only the Hailo repo models I have to run again.
I’ll be happy to provide any additional info to help troubleshoot this or if I just need to be patient for a future package/repo update I can do that too (just please let me know what to wait on and an approximate timeframe)
I cloned main last week (tag 25.3.1) when I did the initial Pi installation
No running processes using /dev/hailo0
Yes, the conf entry for hailo_pci is there
Yes, I had previously enabled PCIe gen 3
I’m currently running the latest kernel from the Raspberry Pi repo: 6.12.25(…)
Since I wrote my original question above, I did some digging and found out about what appears to still be an open issue in terms of 4.19/4.20 of the Hailo packages. I’m not entirely sure if this is still an issue or not as it was raised several months ago. Assuming it is still an open issue, the Raspberry Pi repo is in a somewhat broken state which may be the source of the issue: the DKMS driver is at 4.19 and all the other Hailo packages are at 4.20. If that’s the case, I’ve been looking at how to downgrade the relevant packages to the 4.19 versions (including rpicam-apps* which is giving me some grief but I’d like to keep the camera functionality for testing so I haven’t actually downgraded yet)
I went ahead and downgraded to the 4.19 packages and things are mostly working: I can now run all of the models but I lost the rpicam-apps-hailo-postprocess package in the downgrade. If I can get a compatible version of the postprocess package I should be set until 4.20 is fully functional.
@philb I had issues running the basic detection sample app. Got that resolved by reinstalling the hailo-all package, but after running the pose estimation sample app I got the freeze as well, and even after a reboot the sample apps no longer show video output.
Quick question - what kernel versions are you both running? And did you notice this issue starting to happen around 4.19?
I’m trying to narrow down whether this is a hailort problem, kernel version issue, or something with the apps themselves. My gut feeling is it’s kernel-related since we didn’t see these problems before the 6.12 update, but I want to be sure.
Could you try downgrading to 6.6 and see if the issue goes away? Here’s how to do it:
# 1. Update package lists
sudo apt update
# 2. (Optional) Check available versions to confirm the exact string
apt list -a raspberrypi-kernel raspberrypi-bootloader
# 3. Downgrade/reinstall to 6.6.74+rpt-rpi-2712 (Debian package version 1:6.6.74-1+rpt1)
sudo apt install --reinstall \
raspberrypi-bootloader=1:6.6.74-1+rpt1 \
raspberrypi-kernel=1:6.6.74-1+rpt1
# 4. Hold those two packages so they don't get upgraded
sudo apt-mark hold raspberrypi-bootloader raspberrypi-kernel
# 5. Reboot into the pinned kernel
sudo reboot
After the reboot, you can double-check everything worked with:
uname -a
# Should show "6.6.74+rpt-rpi-2712" and "1:6.6.74-1+rpt1" in the build string
I’m running the latest kernel (6.12.25) and 4.20, with the exception of the DKMS driver which was 4.19, was the first version I installed as it was the current version in the repo. However, after doing further research I found out about the kernel compatibility issues with 4.20. So yesterday I downgraded everything to 4.19 (keeping the Linux kernel and everything else as-is) and that resolved the issue with models hanging etc. So now my only outstanding issue is finding a compatible version of rpicam-apps-hailo-postprocess as that was uninstalled due to the downgrade.
Your post from February ( Critical: HailoRT 4.20 Compatibility Issue with RPi Kernel - Temporary Fix Available ) plainly explained the issue and how to temporarily work around it. It took me a while to come around to the fact that this is still an/the issue. It would probably be helpful if that post were pinned with an explanation that it’s still necessary as I can’t see how any new Raspberry Pi user, like myself last week, won’t run into this. Effectively the Raspberry Pi repos are distributing broken packages for anyone wanting to run the models from the Hailo github repo.
Just a quick followup as I took another look at your February post: it says it’s pinned but does not show up at the top of the posts for me under General (I see the ‘Getting Started with RPI5-Hailo8L’, ‘Raspberry pi 5 with Hailo-8L Benchmark’ and ‘About the General category’ posts pinned but that’s it)
Thanks for pointing that out, but this isn’t the same issue. The HailoRT is stable up until kernel version 6.12.20. We’re working on a fix for this and will update you when we have one ready.