CLIP Zero Shot Inference Application Crashing on Raspberry Pi 5 with Hailo-8L and AI HAT+

General
,
1

Hi everyone,

I’m currently facing an issue with the CLIP Zero Shot Inference Application running on two Raspberry Pi 5 boards (8GB RAM). Each Pi is equipped with a different AI accelerator:

  • One has the Hailo-8L (13 TOPS, HM21LB1C2LAE)
  • The other is using the Raspberry Pi AI HAT+ (26 TOPS)

When I run the command:

python3 clip_application.py --input demo

…the application hangs during inference. It seems like the connection to the Hailo AI accelerator is suddenly lost mid-inference, though nothing unusual appears in the logs.

Here’s what I’ve tried so far:

  • I purchased both AI accelerators at release, but haven’t actively used them until now.
  • Updated the Raspberry Pi OS to the latest version.
  • Performed a clean reinstallation of Raspberry Pi OS.
  • Downgraded the PCIe interface to Gen 2 speeds to avoid potential compatibility issues.
  • Ran hailortcli monitor, which shows the models correctly – but this also suddenly stops updating when the inference crashes.
  • Used dmesg --follow to monitor system logs, but no related messages appear when the application freezes.
  • Measured the heatsink temperature using an infrared (IR) thermometer: around 35–40°C.

Observations:

  • The inference demo should be the simplest test case, yet it still fails.
  • It feels like something is interrupting the connection between the Pi and the Hailo hardware during model execution.

Has anyone experienced something similar, or does anyone have suggestions for deeper debugging steps? These devices are currently hard to use if even the basic demo doesn’t complete. it is recommended to attach a heat sink to the Hailo TPUs for 24/7 use ? I also came across the thread Raspberry Pi Kernel Compatibility Issue - Temporary Fix but unfortunately that didn’t work - it would be interesting to know if there are still problems with it.

Thanks so much for your time and any feedback or ideas you may have!

Best regards,
Matthias

2

Hey @Matthias_Kraft,

Welcome to the Hailo Community!

We haven’t seen any specific issues with that setup yet, but the kernel compatibility problem might affect it as well.

What issues are you running into when trying to downgrade the kernel? Please share what errors or problems you’re seeing and I can help troubleshoot the downgrade process.

3

Thanks for your quick response. I am unable to downgrade the Raspberry Pi kernel to a version compatible with the Hailo AI accelerator, as the required package versions are not found in the pi repos, which prevents me from doing the Downgrade. When I run the demo application, sometimes nothing happens at all. Since I cannot upload a video here, I have attached a link. hailo clip not working.mp4 Maybe this will help to solve the problem. I have also attached a video of the freeze. Hailo Clip Application hangs
Here is the log (same on both Pis):

System Information

matthias@pi5:~/Desktop $ uname -a Linux pi5 6.12.25+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64 GNU/Linux

Available Packages

matthias@pi5:~/Desktop $ apt list -a raspberrypi-kernel raspberrypi-bootloader Listing... Done raspberrypi-bootloader/stable 1:1.20230405-1 arm64 raspberrypi-bootloader/stable 1:1.20230405-1 armhf raspberrypi-kernel/stable 1:1.20230405-1 arm64 raspberrypi-kernel/stable 1:1.20230405-1 armhf

Failed Downgrade Attempts

Attempting to downgrade to version 6.6.74:

`matthias@pi5:~/Desktop $ sudo apt install --reinstall
raspberrypi-bootloader=1:6.6.74-1+rpt1
raspberrypi-kernel=1:6.6.74-1+rpt1

E: Version ‘1:6.6.74-1+rpt1’ for ‘raspberrypi-bootloader’ was not found
E: Version ‘1:6.6.74-1+rpt1’ for ‘raspberrypi-kernel’ was not found`

Attempting to downgrade to version 6.12.20:

`Bashmatthias@pi5:~/Desktop $ sudo apt install --reinstall
raspberrypi-bootloader=1:6.12.20-1+rpt1
raspberrypi-kernel=1:6.12.20-1+rpt1

E: Version ‘1:6.12.20-1+rpt1’ for ‘raspberrypi-bootloader’ was not found
E: Version ‘1:6.12.20-1+rpt1’ for ‘raspberrypi-kernel’ was not found`

I appreciate your suggestions and thank you in advance for your help in resolving this issue.
best regards
Matthias

4

Here is the complete dmesg log from the attempt where the clip application did not start at all.

[ 2.951725] Bluetooth: HCI UART protocol Broadcom registered
[ 2.962876] hailo 0001:01:00.0: File hailo/hailo8_fw.bin written successfully
[ 2.962884] hailo 0001:01:00.0: Writing file hailo/hailo8_board_cfg.bin
[ 2.962914] Failed to write file hailo/hailo8_board_cfg.bin
[ 2.962916] hailo 0001:01:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 2.962918] hailo 0001:01:00.0: Writing file hailo/hailo8_fw_cfg.bin
[ 2.962925] Failed to write file hailo/hailo8_fw_cfg.bin
[ 2.962927] hailo 0001:01:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 3.099520] hailo 0001:01:00.0: NNC Firmware loaded successfully
[ 3.099527] hailo 0001:01:00.0: FW loaded, took 310 ms
[ 3.109693] hailo 0001:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0
[ 3.216973] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 3.218501] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 3.218719] usbcore: registered new interface driver brcmfmac
[ 3.302131] Bluetooth: hci0: BCM: chip id 107
[ 3.302338] Bluetooth: hci0: BCM: features 0x2f
[ 3.303391] Bluetooth: hci0: BCM4345C0
[ 3.303393] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[ 3.331482] Bluetooth: hci0: BCM4345C0 ‘brcm/BCM4345C0.raspberrypi,5-model-b.hcd’ Patch
[ 3.399134] brcmfmac: brcmf_c_process_txcap_blob: no txcap_blob available (err=-2)
[ 3.399459] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Aug 29 2023 01:47:08 version 7.45.265 (28bca26 CY) FWID 01-b677b91b
[ 3.809267] Adding 204784k swap on /var/swap. Priority:-2 extents:9 across:1622016k SS
[ 4.010329] Bluetooth: hci0: BCM: features 0x2f
[ 4.011688] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3±0190
[ 4.011694] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0382
[ 4.011998] Bluetooth: hci0: BCM: Using default device address (43:45:c0:00:1f:ac)
[ 4.100387] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 4.100394] Bluetooth: BNEP filters: protocol multicast
[ 4.100400] Bluetooth: BNEP socket layer initialized
[ 4.102043] Bluetooth: MGMT ver 1.23
[ 4.110763] NET: Registered PF_ALG protocol family
[ 5.040729] macb 1f00100000.ethernet eth0: PHY [1f00100000.ethernet-ffffffff:01] driver [Broadcom BCM54213PE] (irq=POLL)
[ 5.040739] macb 1f00100000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 5.045379] macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
[ 5.058267] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[ 7.743094] Bluetooth: RFCOMM TTY layer initialized
[ 7.743108] Bluetooth: RFCOMM socket layer initialized
[ 7.743118] Bluetooth: RFCOMM ver 1.11
[ 3559.225295] hailo: get_user_pages failed with -4
[ 3559.225306] hailo 0001:01:00.0: failed to set sg list for user buffer -4
[ 3559.225311] hailo 0001:01:00.0: failed map buffer 7ffeeb42c000
[ 3977.966811] hailo: get_user_pages failed with -4
[ 3977.966833] hailo 0001:01:00.0: failed to set sg list for user buffer -4
[ 3977.966839] hailo 0001:01:00.0: failed map buffer 7ffe9406c000
[ 4713.445379] INFO: task clip_inference_:3836 blocked for more than 120 seconds.
[ 4713.445390] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445393] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445396] task:clip_inference_ state:D stack:0 pid:3836 tgid:3783 ppid:3735 flags:0x0000000c
[ 4713.445403] Call trace:
[ 4713.445404] __switch_to+0xf0/0x150
[ 4713.445415] __schedule+0x38c/0xb50
[ 4713.445419] schedule+0x3c/0x148
[ 4713.445424] schedule_preempt_disabled+0x2c/0x50
[ 4713.445429] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445435] down_read+0xac/0xc0
[ 4713.445440] hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
[ 4713.445452] hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
[ 4713.445458] hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
[ 4713.445464] hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
[ 4713.445470] __arm64_sys_ioctl+0xb4/0x100
[ 4713.445476] invoke_syscall+0x50/0x120
[ 4713.445482] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445487] do_el0_svc+0x24/0x38
[ 4713.445492] el0_svc+0x30/0xd0
[ 4713.445495] el0t_64_sync_handler+0x100/0x130
[ 4713.445499] el0t_64_sync+0x190/0x198
[ 4713.445505] INFO: task pool-clip_appli:3849 blocked for more than 120 seconds.
[ 4713.445507] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445510] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445511] task:pool-clip_appli state:D stack:0 pid:3849 tgid:3783 ppid:3735 flags:0x00000004
[ 4713.445516] Call trace:
[ 4713.445517] __switch_to+0xf0/0x150
[ 4713.445522] __schedule+0x38c/0xb50
[ 4713.445526] schedule+0x3c/0x148
[ 4713.445531] schedule_preempt_disabled+0x2c/0x50
[ 4713.445535] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445540] down_read+0xac/0xc0
[ 4713.445545] do_madvise+0x1dc/0xaa8
[ 4713.445548] __arm64_sys_madvise+0x2c/0x48
[ 4713.445551] invoke_syscall+0x50/0x120
[ 4713.445555] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445560] do_el0_svc+0x24/0x38
[ 4713.445564] el0_svc+0x30/0xd0
[ 4713.445567] el0t_64_sync_handler+0x100/0x130
[ 4713.445571] el0t_64_sync+0x190/0x198
[ 4713.445574] INFO: task pool-clip_appli:3850 blocked for more than 120 seconds.
[ 4713.445577] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445579] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445580] task:pool-clip_appli state:D stack:0 pid:3850 tgid:3783 ppid:3735 flags:0x00000004
[ 4713.445584] Call trace:
[ 4713.445586] __switch_to+0xf0/0x150
[ 4713.445590] __schedule+0x38c/0xb50
[ 4713.445595] schedule+0x3c/0x148
[ 4713.445599] schedule_preempt_disabled+0x2c/0x50
[ 4713.445603] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445608] down_read+0xac/0xc0
[ 4713.445612] do_madvise+0x1dc/0xaa8
[ 4713.445615] __arm64_sys_madvise+0x2c/0x48
[ 4713.445618] invoke_syscall+0x50/0x120
[ 4713.445622] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445626] do_el0_svc+0x24/0x38
[ 4713.445631] el0_svc+0x30/0xd0
[ 4713.445634] el0t_64_sync_handler+0x100/0x130
[ 4713.445637] el0t_64_sync+0x190/0x198
[ 4713.445640] INFO: task pool-clip_appli:3851 blocked for more than 120 seconds.
[ 4713.445642] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445644] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445646] task:pool-clip_appli state:D stack:0 pid:3851 tgid:3783 ppid:3735 flags:0x00000004
[ 4713.445649] Call trace:
[ 4713.445651] __switch_to+0xf0/0x150
[ 4713.445655] __schedule+0x38c/0xb50
[ 4713.445659] schedule+0x3c/0x148
[ 4713.445663] schedule_preempt_disabled+0x2c/0x50
[ 4713.445668] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445672] down_read+0xac/0xc0
[ 4713.445677] do_madvise+0x1dc/0xaa8
[ 4713.445679] __arm64_sys_madvise+0x2c/0x48
[ 4713.445682] invoke_syscall+0x50/0x120
[ 4713.445686] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445690] do_el0_svc+0x24/0x38
[ 4713.445695] el0_svc+0x30/0xd0
[ 4713.445698] el0t_64_sync_handler+0x100/0x130
[ 4713.445701] el0t_64_sync+0x190/0x198
[ 4713.445704] INFO: task pool-clip_appli:3852 blocked for more than 120 seconds.
[ 4713.445706] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445708] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445710] task:pool-clip_appli state:D stack:0 pid:3852 tgid:3783 ppid:3735 flags:0x00000004
[ 4713.445713] Call trace:
[ 4713.445715] __switch_to+0xf0/0x150
[ 4713.445719] __schedule+0x38c/0xb50
[ 4713.445723] schedule+0x3c/0x148
[ 4713.445727] schedule_preempt_disabled+0x2c/0x50
[ 4713.445731] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445736] down_read+0xac/0xc0
[ 4713.445741] do_madvise+0x1dc/0xaa8
[ 4713.445743] __arm64_sys_madvise+0x2c/0x48
[ 4713.445746] invoke_syscall+0x50/0x120
[ 4713.445750] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445754] do_el0_svc+0x24/0x38
[ 4713.445759] el0_svc+0x30/0xd0
[ 4713.445762] el0t_64_sync_handler+0x100/0x130
[ 4713.445765] el0t_64_sync+0x190/0x198
[ 4713.445768] INFO: task pool-clip_appli:3853 blocked for more than 120 seconds.
[ 4713.445771] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4713.445773] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.445774] task:pool-clip_appli state:D stack:0 pid:3853 tgid:3783 ppid:3735 flags:0x00000004
[ 4713.445778] Call trace:
[ 4713.445779] __switch_to+0xf0/0x150
[ 4713.445783] __schedule+0x38c/0xb50
[ 4713.445788] schedule+0x3c/0x148
[ 4713.445791] schedule_preempt_disabled+0x2c/0x50
[ 4713.445796] rwsem_down_read_slowpath+0x214/0x500
[ 4713.445800] down_read+0xac/0xc0
[ 4713.445805] do_madvise+0x1dc/0xaa8
[ 4713.445807] arm64_sys_madvise+0x2c/0x48
[ 4713.445810] invoke_syscall+0x50/0x120
[ 4713.445814] el0_svc_common.constprop.0+0x48/0xf0
[ 4713.445819] do_el0_svc+0x24/0x38
[ 4713.445823] el0_svc+0x30/0xd0
[ 4713.445826] el0t_64_sync_handler+0x100/0x130
[ 4713.445829] el0t_64_sync+0x190/0x198
[ 4834.277573] INFO: task clip_inference:3836 blocked for more than 241 seconds.
[ 4834.277584] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4834.277588] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4834.277590] task:clip_inference state:D stack:0 pid:3836 tgid:3783 ppid:3735 flags:0x0000000c
[ 4834.277597] Call trace:
[ 4834.277599] __switch_to+0xf0/0x150
[ 4834.277611] __schedule+0x38c/0xb50
[ 4834.277615] schedule+0x3c/0x148
[ 4834.277620] schedule_preempt_disabled+0x2c/0x50
[ 4834.277625] rwsem_down_read_slowpath+0x214/0x500
[ 4834.277631] down_read+0xac/0xc0
[ 4834.277637] hailo_vdma_buffer_map+0x3c4/0x6a0 [hailo_pci]
[ 4834.277651] hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
[ 4834.277657] hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
[ 4834.277664] hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
[ 4834.277670] __arm64_sys_ioctl+0xb4/0x100
[ 4834.277676] invoke_syscall+0x50/0x120
[ 4834.277683] el0_svc_common.constprop.0+0x48/0xf0
[ 4834.277688] do_el0_svc+0x24/0x38
[ 4834.277693] el0_svc+0x30/0xd0
[ 4834.277697] el0t_64_sync_handler+0x100/0x130
[ 4834.277701] el0t_64_sync+0x190/0x198
[ 4834.277706] INFO: task pool-clip_appli:3849 blocked for more than 241 seconds.
[ 4834.277709] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4834.277711] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4834.277713] task:pool-clip_appli state:D stack:0 pid:3849 tgid:3783 ppid:3735 flags:0x00000004
[ 4834.277718] Call trace:
[ 4834.277720] __switch_to+0xf0/0x150
[ 4834.277725] __schedule+0x38c/0xb50
[ 4834.277729] schedule+0x3c/0x148
[ 4834.277734] schedule_preempt_disabled+0x2c/0x50
[ 4834.277738] rwsem_down_read_slowpath+0x214/0x500
[ 4834.277744] down_read+0xac/0xc0
[ 4834.277748] do_madvise+0x1dc/0xaa8
[ 4834.277753] __arm64_sys_madvise+0x2c/0x48
[ 4834.277756] invoke_syscall+0x50/0x120
[ 4834.277760] el0_svc_common.constprop.0+0x48/0xf0
[ 4834.277765] do_el0_svc+0x24/0x38
[ 4834.277770] el0_svc+0x30/0xd0
[ 4834.277773] el0t_64_sync_handler+0x100/0x130
[ 4834.277777] el0t_64_sync+0x190/0x198
[ 4834.277780] INFO: task pool-clip_appli:3850 blocked for more than 241 seconds.
[ 4834.277782] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4834.277784] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4834.277786] task:pool-clip_appli state:D stack:0 pid:3850 tgid:3783 ppid:3735 flags:0x00000004
[ 4834.277790] Call trace:
[ 4834.277792] __switch_to+0xf0/0x150
[ 4834.277796] __schedule+0x38c/0xb50
[ 4834.277801] schedule+0x3c/0x148
[ 4834.277805] schedule_preempt_disabled+0x2c/0x50
[ 4834.277810] rwsem_down_read_slowpath+0x214/0x500
[ 4834.277815] down_read+0xac/0xc0
[ 4834.277820] do_madvise+0x1dc/0xaa8
[ 4834.277823] __arm64_sys_madvise+0x2c/0x48
[ 4834.277826] invoke_syscall+0x50/0x120
[ 4834.277830] el0_svc_common.constprop.0+0x48/0xf0
[ 4834.277835] do_el0_svc+0x24/0x38
[ 4834.277840] el0_svc+0x30/0xd0
[ 4834.277843] el0t_64_sync_handler+0x100/0x130
[ 4834.277846] el0t_64_sync+0x190/0x198
[ 4834.277849] INFO: task pool-clip_appli:3851 blocked for more than 241 seconds.
[ 4834.277852] Not tainted 6.12.25+rpt-rpi-2712 #1 Debian 1:6.12.25-1+rpt1
[ 4834.277854] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4834.277856] task:pool-clip_appli state:D stack:0 pid:3851 tgid:3783 ppid:3735 flags:0x00000004
[ 4834.277860] Call trace:
[ 4834.277862] __switch_to+0xf0/0x150
[ 4834.277866] __schedule+0x38c/0xb50
[ 4834.277871] schedule+0x3c/0x148
[ 4834.277875] schedule_preempt_disabled+0x2c/0x50
[ 4834.277880] rwsem_down_read_slowpath+0x214/0x500
[ 4834.277885] down_read+0xac/0xc0
[ 4834.277890] do_madvise+0x1dc/0xaa8
[ 4834.277892] __arm64_sys_madvise+0x2c/0x48
[ 4834.277895] invoke_syscall+0x50/0x120
[ 4834.277900] el0_svc_common.constprop.0+0x48/0xf0
[ 4834.277904] do_el0_svc+0x24/0x38
[ 4834.277909] el0_svc+0x30/0xd0
[ 4834.277912] el0t_64_sync_handler+0x100/0x130
[ 4834.277916] el0t_64_sync+0x190/0x198
[ 4834.277919] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

5

Hi,
to downgrade the kernel to 6.6.78 try this:
sudo rpi-update 192d8e0b9d

Here are a list of some other kernel (6.12.x + 6.6.x ) hashes:
[BUG] Deadlock in hailo_pci driver on Raspberry Pi 5 (Kernel 6.12.x) - #11 by Kletternaut?

1 Like