Problems running the hailo raspi5 examples

General
1

Hi,

My Setup:
Hailo8l based AI-HAT+
Raspi 5, 4 GB RAM
Raspberry Pi OS 64 bit, up to date (06.05.25)

Two months ago I installed the hailo rpi5 examples on my raspi and I could execute all the “hailo-rpi5-examples” without problems. Now I have the problem that only the detection_simple.py example is running sometimes.
All other examples will start (and have some output), but the preview video does not appear and the inferencing does not start.

Journalctl -f shows me the following lines which may cause the problem:

May 06 10:31:03 raspikatze kernel: hailo: get_user_pages failed with -4
May 06 10:31:03 raspikatze kernel: hailo 0001:01:00.0: failed to set sg list for user buffer -4
May 06 10:31:03 raspikatze kernel: hailo 0001:01:00.0: failed map buffer 7fff20428000

If the detection_simple.py examples runs successfully I will not see these lines in the kernel log.
I made a lot of testing on my OS installation and so I took a brand new SD card and installed a fresh copy of the latest Raspberry Pi OS 64 bit. Then:

sudo apt update
sudo apt full-upgrade
sudo apt install hailo-all
reboot
git clone GitHub - hailo-ai/hailo-rpi5-examples
./install.sh
source setup_env.sh
python basic_pipelines/detection.py

Unfortunately same behavior as before. Nothing else was installed and so it looks like some kind of driver problem.

hailortcli fw-control identify:
Executing on device: 0001:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.20.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8L
Serial Number: <N/A>
Part Number: <N/A>
Product Name: <N/A>

sudo rpi-eeprom-update
BOOTLOADER: up to date
CURRENT: Wed 19 Mar 13:41:26 UTC 2025 (1742391686)
LATEST: Mon 10 Mar 17:10:37 UTC 2025 (1741626637)
RELEASE: default (/usr/lib/firmware/raspberrypi/bootloader-2712/default)

As this is so easy to reproduce it may affect not only me.

Any suggestions?

2

I experienced the same issue after the latest update. I’m not entirely sure what the exact cause is, but I suspect the new kernel version 6.12.25 might be to blame. I restored a backup image with version 6.12.20, and everything is working fine now.

3

This is the update history log for the change that broke the system:
Start-Date: 2025-05-04 21:23:20

Commandline: packagekit role=‘update-packages’

Requested-By: User (1000)

Install: linux-headers-6.12.25+rpt-rpi-v8:arm64 (1:6.12.25-1+rpt1, automatic), linux-headers-6.12.25+rpt-common-rpi:arm64 (1:6.12.25-1+rpt1, automatic), linux-image-6.12.25+rpt-rpi-2712:arm64 (1:6.12.25-1+rpt1, automatic), linux-image-6.12.25+rpt-rpi-v8:arm64 (1:6.12.25-1+rpt1, automatic), linux-headers-6.12.25+rpt-rpi-2712:arm64 (1:6.12.25-1+rpt1, automatic), libcamera0.5:arm64 (0.5.0+rpt20250429-1, automatic), linux-kbuild-6.12.25+rpt:arm64 (1:6.12.25-1+rpt1, automatic)

Upgrade: libpipewire-0.3-common:arm64 (1.2.7-1bpo12+1+rpt3), pipewire-pulse:arm64 (1.2.7-1bpo12+1+rpt3), pipanel:arm64 (1:1.52, 1:1.53), libspa-0.2-libcamera:arm64 (1.2.7-1bpo12+1+rpt3), openjdk-17-jre:arm64 (17.0.14+7-1deb12u1), libpisp1:arm64 (1.1.0-1, 1.2.1-1), linux-headers-rpi-v8:arm64 (1:6.12.20-1+rpt1bpo12+1+rpt2, 1.2.7-1bpo12+1, 1:6.12.25-1+rpt1), libcamera-ipa:arm64 (0.4.0+rpt20250213-1, 0.5.0+rpt20250429-1), linux-image-rpi-2712:arm64 (1:6.12.20-1+rpt1bpo12+1+rpt2, 1.2.7-1bpo12+1+rpt2, 1.2.7-1bpo12+1+rpt2, 1.2.7-1bpo12+1, 1:6.12.25-1+rpt1), libspa-0.2-modules:arm64 (1.2.7-1bpo12+1+rpt3), libpipewire-0.3-modules:arm64 (1.2.7-1bpo12+1+rpt3), python3-libcamera:arm64 (0.4.0+rpt20250213-1, 0.5.0+rpt20250429-1), rpicam-apps:arm64 (1.6.0-2, 1.7.0-1), openjdk-17-jre-headless:arm64 (17.0.14+7-1deb12u1), rpicam-apps-hailo-postprocess:arm64 (1.6.0-2, 1.7.0-1), linux-libc-dev:arm64 (1:6.12.20-1+rpt1bpo12+1+rpt2, 1.2.7-1~bpo12+1+rpt3)

End-Date: 2025-05-04 21:24:54

4

Hmm, this looks like a real show stopper on the Raspi.
But I’m lucky, I only want to realize a hobby project, so I’ll wait for a solution before going on testing…

1 Like
5

My Setup:
Hailo8l based AI-HAT
Raspi 5, 16 GB RAM

Also was working great, hailo-rpi5-examples worked without problems. Also while trying to resolve, I started with a new SD and installed latest Raspberry PI OS 64 bit and went thru the Hailo install as described. hailo-rpi5-examples do not work, no video output. I am able to execute if I use --input rpi, but still not as stable as previously. Also, when I try to use --use-frame, the code will run for a few minutes, then the shell hangs, unable to control-c to break out of the execution.

6

I have been having Hailo lockup issues ever sense the 6.12.25 Linux update on the RPi. Seems like something is not letting go at the kernel level. Even sudo modprob and ps cmds lock once the issue occurs. Only solution is to reboot. my code, the rpi examples and even degirum code all see same lockup issue.

7

Hi @reddtedd59
Can you tell us what happens after the reboot? There is lockup again and you need to reboot again?

8

each run of any of the apps works one time, then the second time it locks up. I see the following hailo related messages in the system log:
May 18 12:08:58 rp6 kernel: INFO: task inference_hailo:48400 blocked for more than 120 seconds.
May 18 12:08:58 rp6 kernel: task:inference_hailo state:D stack:0 pid:48400 tgid:48322 ppid:48321 flags:0x0000000c
May 18 12:08:58 rp6 kernel: hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
May 18 12:08:58 rp6 kernel: hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
May 18 12:08:58 rp6 kernel: hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
May 18 12:08:58 rp6 kernel: hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
May 18 12:10:59 rp6 kernel: INFO: task inference_hailo:48400 blocked for more than 241 seconds.
May 18 12:10:59 rp6 kernel: task:inference_hailo state:D stack:0 pid:48400 tgid:48322 ppid:48321 flags:0x0000000c
May 18 12:10:59 rp6 kernel: hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
May 18 12:10:59 rp6 kernel: hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
May 18 12:10:59 rp6 kernel: hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
May 18 12:10:59 rp6 kernel: hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
May 18 12:12:15 rp6 sudo[56946]: ted : TTY=pts/1 ; PWD=/home/ted/iCodeLocal/hailoWork/Monitor ; USER=root ; COMMAND=/usr/sbin/modprobe -r hailo_pci
May 18 12:13:00 rp6 kernel: INFO: task inference_hailo:48400 blocked for more than 362 seconds.
May 18 12:13:00 rp6 kernel: task:inference_hailo state:D stack:0 pid:48400 tgid:48322 ppid:48321 flags:0x0000000c
May 18 12:13:00 rp6 kernel: hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
May 18 12:13:00 rp6 kernel: hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
May 18 12:13:00 rp6 kernel: hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
May 18 12:13:00 rp6 kernel: hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
May 18 12:13:09 rp6 kernel: hailo: get_user_pages failed with -4
May 18 12:13:09 rp6 kernel: hailo 0001:01:00.0: failed to set sg list for user buffer -4
May 18 12:13:09 rp6 kernel: hailo 0001:01:00.0: failed map buffer 7ffec73d8000
May 18 12:13:14 rp6 sudo[58131]: ted : TTY=pts/1 ; PWD=/home/ted/iCodeLocal/hailoWork/Monitor ; USER=root ; COMMAND=/usr/sbin/modprobe -r hailo_pci
May 18 12:13:14 rp6 kernel: hailo: Exit module.
May 18 12:13:14 rp6 kernel: hailo 0001:01:00.0: Remove: Releasing board
May 18 12:13:14 rp6 kernel: hailo 0001:01:00.0: Remove: Freed board, /dev/hailo0
May 18 12:13:14 rp6 kernel: hailo: Hailo PCIe driver unloaded.

I killed task 48400 and it brought the hailo accerator back to life for one more inference run
inference_hailo:48400 blocked for more than 362 seconds

9

Hey All,

Checking in , Is the issue still persists ? If so , Can you try and run for now the tag 25.3.1 for now , we are working on updating our whole repo packages for better usage and development.

10

just retested and problem is still there:
May 22 12:34:49 rp6 kernel: INFO: task inference_hailo:4602 blocked for more than 241 seconds.
May 22 12:34:49 rp6 kernel: task:inference_hailo state:D stack:0 pid:4602 tgid:4524 ppid:4523 flags:0x00000004

I am running:
hailo-all/stable,stable,now 4.20.0 all [installed]
Hailo support (metapackage)

hailo-dkms/stable,stable 4.19.0-1 all
Hailo PCIe driver

hailo-tappas-core/stable,now 3.31.0+1-1 arm64 [installed,automatic]
This package contains the core components of the Hailo Tappas platform.

hailo-tappas-core-3.28.2/stable 3.28.2 arm64
This package contains the core components of the Hailo Tappas platform.

hailofw/stable,stable,now 4.20.0-1 all [installed,automatic]
Hailo firmware

hailort/stable,now 4.20.0-1 arm64 [installed,automatic]
HailoRT

libbot-training-perl/stable,stable 0.07-4 all
text training material for bots

python3-hailort/stable,now 4.20.0-1 arm64 [installed,automatic]
HailoRT Python API, which wraps the runtime library

do you want me to sudo apt -t 25.3.1 upgrade hailo-all?

11

had some time and extra parts so I did a clean build on a new RPI5 and this time using a hailo8 NPU instead of a hailo8l. all software installed fresh and updated. same exact issue running detection.py. will run once, then hangs on second try. reboot required to free it back up.

is there hope for a solution soon? now two rpis and two hailo ai kits…

12

Hey All,

Is this issue happening only on video, or does it get one detection on camera and then stop?

Also, does this happen in a loop - like when the video plays the second time, or do the detections just stop after a few seconds of running?

I was only able to reproduce this issue on kernel 6.12 with video loop. Got it solved by just going back to 6.6 - that issue is being checked!

I personally work on 6.6.74 and it’s pretty stable.

Can you try the following?

Run:

sudo lsof /dev/hailo0

If you see any processes, kill them:

sudo kill -9 <PID>

Apply PCIe page size override:
Make sure /etc/modprobe.d/hailo_pci.conf exists and contains:

options hailo_pci force_desc_page_size=4096

Then reboot and retry.

If it’s still not working, I’d suggest downgrading the kernel might help - but don’t go under 6.6.

Let me know if this helps or what errors you face !

13

Hi Adrian, could you explain me how to proceed with the downgrade? I am facing the same issue with kernel 6.12.25 and i would like to downgrade to 6.12.20

14

Hey @Paloma_Jimeno ,

Welcome to the Hailo Community!

This is how to downgrade to 6.6 kernel :

# 1. Update package lists
sudo apt update
# 2. (Optional) Check available versions to confirm the exact string
apt list -a raspberrypi-kernel raspberrypi-bootloader
# 3. Downgrade/reinstall to 6.6.74+rpt-rpi-2712 (Debian package version 1:6.6.74-1+rpt1)
sudo apt install --reinstall \
  raspberrypi-bootloader=1:6.6.74-1+rpt1 \
  raspberrypi-kernel=1:6.6.74-1+rpt1
# 4. Hold those two packages so they don't get upgraded
sudo apt-mark hold raspberrypi-bootloader raspberrypi-kernel
# 5. Reboot into the pinned kernel
sudo reboot

After the reboot, you can double-check everything worked with:

uname -a
# Should show "6.6.74+rpt-rpi-2712" and "1:6.6.74-1+rpt1" in the build string

Let me know what you find!

15

Thank you. But my current kernel is 1:6.12.25-1+rpt1 and i read here that the problem I suffer with hailo examples is fixed when downgrade to kernel 6.12.20. So i would like to have this kernel version

16

I’m having a very similar problem. I had the problem on my rpi5 with 8g and then tried it on a new 16gb rpi5. I tried editing the hailo_pci.conf file but that did not seem to work. The detection.py sample seems to work, but the pose_estimation.py demo does not run, on either the sample video mp4 file or “–input rpi”. When I use my USB camera via “–input usb” camera it runs for a few seconds and then fails with “get_user_pages failed with -4” error. I am encouraged that Adrian had success by downgrading kernel and would like to try to downgrade the kernel to 6.12.20. How do you do it? I’d rather not go back to 6.6.74 kernel, if I don’t have to.

17

I was able to downgrade to 6.12.20. You apparently can’t use “apt install -reinstall” to bump down to 6.12.20. I ran “apt list -a” to confirm that I had 6.12.20 in my repository.

sudo apt install \
  linux-image-6.12.20+rpt-rpi-2712=1:6.12.20-1+rpt1~bpo12+1 \
  linux-headers-6.12.20+rpt-rpi-2712=1:6.12.20-1+rpt1~bpo12+1

I located the files inside /boot/
vmlinuz-6.12.20+rpt-rpi-2712
initrd.img-6.12.20+rpt-rpi-2712

and copied these into /boot/firmware

Then I edited /boot/firmware/config.txt to add the following lines:

kernel=vmlinuz-6.12.20+rpt-rpi-2712
initramfs initrd.img-6.12.20+rpt-rpi-2712 followkernel


sudo cp /boot/vmlinuz-6.12.20+rpt-rpi-2712 /boot/firmware/kernel8.img
sudo reboot

after confirm running kernel with uname -a.

After I did this, my pose_estimation.py demo works and seems pretty stable with video files, --input rpi and --input usb!

Hope this helps.
-Ira

1 Like
18

Hi Ira,

Thank you for your answer. The downgrade was successful and also, all the hailo-rpi5-examples ran perfectly well. Thank you for your support!

19

I am also successfully running after manually down grading the kernel to 6.12.20+rpt-rpi-2712. I have done a upgrade of the hailo-all to latest version. Issue seems to be with an interaction between hailo and the 6.12.25 kernel. I am running my inferences against video .mp4
files.