After upgrading my Raspberry Pi 5 to kernel 6.12.x, I am experiencing a deadlock in the hailo_pci
driver when running my custom pose estimation script. The system hangs after a few seconds. The official Hailo demos still work without issues. This problem did not occur with previous kernel versions.
- Hardware: Raspberry Pi 5, Hailo8 PCIe
- Kernel: 6.12.x (e.g. 6.12.30-v8-16k+)
/etc/modprobe.d/hailo_pci.conf
is present- The official Hailo demos run without issues
==== Hailo Bugreport Info ====
Datum: Mi 4. Jun 13:47:09 CEST 2025
== Hailo SDK/Tools Version ==
HailoRT-CLI version 4.20.0
ii hailo-all 4.20.0 all Hailo support (metapackage)
ii hailo-tappas-core 3.31.0+1-1 arm64 This package contains the core components of the Hailo Tappas platform.
ii hailofw 4.20.0-1 all Hailo firmware
ii hailort 4.20.0-1 arm64 HailoRT
ii python3-hailort 4.20.0-1 arm64 HailoRT Python API, which wraps the runtime library
ii rpicam-apps-hailo-postprocess 1.7.0-1 arm64 rpicam-apps-hailo
== PCIe Info ==
0001:03:00.0 Co-processor [0b40]: Hailo Technologies Ltd. Hailo-8 AI Processor [1e60:2864] (rev 01)
== Kernel & System ==
Linux raspi5 6.12.30-v8-16k+ #1879 SMP PREEMPT Fri May 23 13:30:44 BST 2025 aarch64 GNU/Linux
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
== Hailo Treiber Status ==
hailo_pci 131072 2
filename: /lib/modules/6.12.30-v8-16k+/kernel/drivers/media/pci/hailo/hailo_pci.ko.xz
version: 4.20.0
license: GPL v2
description: Hailo PCIe driver
author: Hailo Technologies Ltd.
import_ns: DMA_BUF
srcversion: E8ED835F8D6D454EABB6BF8
alias: pci:v00001E60d000043A2sv*sd*bc*sc*i*
alias: pci:v00001E60d000045C4sv*sd*bc*sc*i*
alias: pci:v00001E60d00002864sv*sd*bc*sc*i*
depends:
intree: Y
name: hailo_pci
vermagic: 6.12.30-v8-16k+ SMP preempt mod_unload modversions aarch64
parm: o_dbg:int
parm: no_power_mode:Disables automatic D0->D3 PCIe transactions (invbool)
parm: force_allocation_from_driver:Determines whether to force buffer allocation from driver or userspace (int)
parm: force_desc_page_size:Determines the maximum DMA descriptor page size (must be a power of 2) (int)
parm: force_hailo10h_legacy_mode:Forces work with Hailo10h in legacy mode(relevant for emulators) (bool)
parm: force_boot_linux_from_eemc:Boot the linux image from eemc (Requires special Image) (bool)
parm: support_soft_reset:enables driver reload to reload a new firmware as well (bool)
== Hailo PCIe dmesg ==
[ 1.891557] hailo: Init module. driver version 4.20.0
[ 1.891641] hailo 0001:03:00.0: Probing on: 1e60:2864...
[ 1.891644] hailo 0001:03:00.0: Probing: Allocate memory for device extension, 13184
[ 1.891696] hailo 0001:03:00.0: enabling device (0000 -> 0002)
[ 1.891704] hailo 0001:03:00.0: Probing: Device enabled
[ 1.891718] hailo 0001:03:00.0: Probing: mapped bar 0 - 0000000000b8df9c 16384
[ 1.891722] hailo 0001:03:00.0: Probing: mapped bar 2 - 0000000028d70111 4096
[ 1.891725] hailo 0001:03:00.0: Probing: mapped bar 4 - 00000000b7fad923 16384
[ 1.891728] hailo 0001:03:00.0: Probing: Force setting max_desc_page_size to 4096 (recommended value is 16384)
[ 1.891735] hailo 0001:03:00.0: Probing: Enabled 64 bit dma
[ 1.891738] hailo 0001:03:00.0: Probing: Using userspace allocated vdma buffers
[ 1.891742] hailo 0001:03:00.0: Disabling ASPM L0s
[ 1.891746] hailo 0001:03:00.0: Successfully disabled ASPM L0s
[ 1.897740] hailo 0001:03:00.0: Writing file hailo/hailo8_fw.bin
[ 1.989079] hailo 0001:03:00.0: File hailo/hailo8_fw.bin written successfully
[ 1.989089] hailo 0001:03:00.0: Writing file hailo/hailo8_board_cfg.bin
[ 1.989118] Failed to write file hailo/hailo8_board_cfg.bin
[ 1.989120] hailo 0001:03:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 1.989122] hailo 0001:03:00.0: Writing file hailo/hailo8_fw_cfg.bin
[ 1.989128] Failed to write file hailo/hailo8_fw_cfg.bin
[ 1.989130] hailo 0001:03:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 2.080487] hailo 0001:03:00.0: NNC Firmware loaded successfully
[ 2.080495] hailo 0001:03:00.0: FW loaded, took 182 ms
[ 2.093684] hailo 0001:03:00.0: Probing: Added board 1e60-2864, /dev/hailo0
[ 9305.083002] INFO: task Hailo Pose Esti:36253 blocked for more than 120 seconds.
[ 9305.083013] task:Hailo Pose Esti state:D stack:0 pid:36253 tgid:36235 ppid:23020 flags:0x00000004
[ 9305.083042] hailo_vdma_buffer_map+0x3c8/0x6b0 [hailo_pci]
[ 9305.083050] hailo_vdma_buffer_map_ioctl+0xcc/0x348 [hailo_pci]
[ 9305.083054] hailo_vdma_ioctl+0x254/0x2a8 [hailo_pci]
[ 9305.083058] hailo_pcie_fops_unlockedioctl+0x16c/0x7e8 [hailo_pci]
[ 9425.915267] INFO: task Hailo Pose Esti:36253 blocked for more than 241 seconds.
[ 9425.915282] task:Hailo Pose Esti state:D stack:0 pid:36253 tgid:36235 ppid:23020 flags:0x00000004
[ 9425.915324] hailo_vdma_buffer_map+0x3c8/0x6b0 [hailo_pci]
[ 9425.915335] hailo_vdma_buffer_map_ioctl+0xcc/0x348 [hailo_pci]
[ 9425.915341] hailo_vdma_ioctl+0x254/0x2a8 [hailo_pci]
[ 9425.915347] hailo_pcie_fops_unlockedioctl+0x16c/0x7e8 [hailo_pci]
[ 9526.980424] hailo: get_user_pages failed with -4
[ 9526.980434] hailo 0001:03:00.0: failed to set sg list for user buffer -4
[ 9526.980438] hailo 0001:03:00.0: failed map buffer 7fff5015c000
== Aktuelle Kernel-Fehler (dmesg) ==
[ 261.412971] rp1-cfe 1f00110000.csi: Using a link rate of 900 Mbps
[ 279.478745] usb 3-1: new full-speed USB device number 3 using xhci-hcd
[ 279.637450] usb 3-1: New USB device found, idVendor=1a86, idProduct=7523, bcdDevice= 2.64
[ 279.637458] usb 3-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 279.637461] usb 3-1: Product: USB Serial
[ 279.697864] usbcore: registered new interface driver usbserial_generic
[ 279.697879] usbserial: USB Serial support registered for generic
[ 279.699584] usbcore: registered new interface driver ch341
[ 279.699626] usbserial: USB Serial support registered for ch341-uart
[ 279.699675] ch341 3-1:1.0: ch341-uart converter detected
[ 279.712528] usb 3-1: ch341-uart converter now attached to ttyUSB0
[ 300.716714] rp1-cfe 1f00110000.csi: Using a link rate of 900 Mbps
[ 304.572600] rp1-cfe 1f00110000.csi: Using a link rate of 900 Mbps
[ 323.911856] rp1-cfe 1f00110000.csi: Using a link rate of 900 Mbps
[ 9098.546669] rp1-cfe 1f00110000.csi: Using a link rate of 900 Mbps
[ 9305.083002] INFO: task Hailo Pose Esti:36253 blocked for more than 120 seconds.
[ 9305.083010] Tainted: G C 6.12.30-v8-16k+ #1879
[ 9305.083012] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.083013] task:Hailo Pose Esti state:D stack:0 pid:36253 tgid:36235 ppid:23020 flags:0x00000004
[ 9305.083018] Call trace:
[ 9305.083019] __switch_to+0xf0/0x160
[ 9305.083027] __schedule+0x32c/0xb60
[ 9305.083031] schedule+0x3c/0x148
[ 9305.083034] schedule_preempt_disabled+0x2c/0x50
[ 9305.083037] rwsem_down_read_slowpath+0x1e0/0x510
[ 9305.083040] down_read+0xb0/0xc0
[ 9305.083042] hailo_vdma_buffer_map+0x3c8/0x6b0 [hailo_pci]
[ 9305.083050] hailo_vdma_buffer_map_ioctl+0xcc/0x348 [hailo_pci]
[ 9305.083054] hailo_vdma_ioctl+0x254/0x2a8 [hailo_pci]
[ 9305.083058] hailo_pcie_fops_unlockedioctl+0x16c/0x7e8 [hailo_pci]
[ 9305.083062] __arm64_sys_ioctl+0xb0/0x100
[ 9305.083066] invoke_syscall+0x50/0x120
[ 9305.083071] el0_svc_common.constprop.0+0x48/0xf8
[ 9305.083075] do_el0_svc+0x28/0x40
[ 9305.083077] el0_svc+0x30/0xd0
[ 9305.083080] el0t_64_sync_handler+0x13c/0x158
[ 9305.083083] el0t_64_sync+0x190/0x198
[ 9305.083089] INFO: task pool-python:36297 blocked for more than 120 seconds.
[ 9305.083091] Tainted: G C 6.12.30-v8-16k+ #1879
[ 9305.083092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9305.083093] task:pool-python state:D stack:0 pid:36297 tgid:36235 ppid:23020 flags:0x00000004
[ 9305.083096] Call trace:
[ 9305.083097] __switch_to+0xf0/0x160
[ 9305.083101] __schedule+0x32c/0xb60
[ 9305.083104] schedule+0x3c/0x148
[ 9305.083107] schedule_preempt_disabled+0x2c/0x50
[ 9305.083110] rwsem_down_read_slowpath+0x1e0/0x510
[ 9305.083112] down_read+0xb0/0xc0
[ 9305.083114] do_madvise.part.0+0x190/0xad8
[ 9305.083117] __arm64_sys_madvise+0x54/0x78
[ 9305.083119] invoke_syscall+0x50/0x120
[ 9305.083122] el0_svc_common.constprop.0+0x48/0xf8
[ 9305.083125] do_el0_svc+0x28/0x40
[ 9305.083128] el0_svc+0x30/0xd0
[ 9305.083130] el0t_64_sync_handler+0x13c/0x158
[ 9305.083133] el0t_64_sync+0x190/0x198
[ 9425.915267] INFO: task Hailo Pose Esti:36253 blocked for more than 241 seconds.
[ 9425.915277] Tainted: G C 6.12.30-v8-16k+ #1879
[ 9425.915280] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9425.915282] task:Hailo Pose Esti state:D stack:0 pid:36253 tgid:36235 ppid:23020 flags:0x00000004
[ 9425.915289] Call trace:
[ 9425.915291] __switch_to+0xf0/0x160
[ 9425.915301] __schedule+0x32c/0xb60
[ 9425.915307] schedule+0x3c/0x148
[ 9425.915312] schedule_preempt_disabled+0x2c/0x50
[ 9425.915317] rwsem_down_read_slowpath+0x1e0/0x510
[ 9425.915321] down_read+0xb0/0xc0
[ 9425.915324] hailo_vdma_buffer_map+0x3c8/0x6b0 [hailo_pci]
[ 9425.915335] hailo_vdma_buffer_map_ioctl+0xcc/0x348 [hailo_pci]
[ 9425.915341] hailo_vdma_ioctl+0x254/0x2a8 [hailo_pci]
[ 9425.915347] hailo_pcie_fops_unlockedioctl+0x16c/0x7e8 [hailo_pci]
[ 9425.915353] __arm64_sys_ioctl+0xb0/0x100
[ 9425.915359] invoke_syscall+0x50/0x120
[ 9425.915366] el0_svc_common.constprop.0+0x48/0xf8
[ 9425.915371] do_el0_svc+0x28/0x40
[ 9425.915375] el0_svc+0x30/0xd0
[ 9425.915380] el0t_64_sync_handler+0x13c/0x158
[ 9425.915384] el0t_64_sync+0x190/0x198
[ 9425.915392] INFO: task pool-python:36297 blocked for more than 241 seconds.
[ 9425.915395] Tainted: G C 6.12.30-v8-16k+ #1879
[ 9425.915397] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9425.915398] task:pool-python state:D stack:0 pid:36297 tgid:36235 ppid:23020 flags:0x00000004
[ 9425.915403] Call trace:
[ 9425.915405] __switch_to+0xf0/0x160
[ 9425.915410] __schedule+0x32c/0xb60
[ 9425.915415] schedule+0x3c/0x148
[ 9425.915420] schedule_preempt_disabled+0x2c/0x50
[ 9425.915425] rwsem_down_read_slowpath+0x1e0/0x510
[ 9425.915428] down_read+0xb0/0xc0
[ 9425.915431] do_madvise.part.0+0x190/0xad8
[ 9425.915434] __arm64_sys_madvise+0x54/0x78
[ 9425.915438] invoke_syscall+0x50/0x120
[ 9425.915442] el0_svc_common.constprop.0+0x48/0xf8
[ 9425.915447] do_el0_svc+0x28/0x40
[ 9425.915451] el0_svc+0x30/0xd0
[ 9425.915455] el0t_64_sync_handler+0x13c/0x158
[ 9425.915459] el0t_64_sync+0x190/0x198
[ 9526.980424] hailo: get_user_pages failed with -4
[ 9526.980434] hailo 0001:03:00.0: failed to set sg list for user buffer -4
[ 9526.980438] hailo 0001:03:00.0: failed map buffer 7fff5015c000
== /etc/modprobe.d/hailo_pci.conf ==
options hailo_pci force_desc_page_size=4096