Raspberry Pi Kernel Compatibility Issue - Temporary Fix

Hi everyone!

UPDATE: Issue Resolved


This issue has been resolved in Raspberry Pi kernel version 6.12.34.

To update your system to the latest kernel, run the following commands:

sudo apt update
sudo apt full-upgrade
sudo reboot

After the reboot, you can verify the new kernel version by running:

uname -r

We’ve been getting reports about problems running Raspberry Pi examples after recent kernel updates. We’re aware of this issue and actively working on a permanent solution.

Quick Fix: For now, the easiest workaround is to downgrade your kernel to version 6.12.20 or earlier (or any 6.6.x version). This should get everything working smoothly again.

How to Downgrade Your Raspberry Pi Kernel

Before running this backup your Pi to prevent data loss

Downgrade to kernel 6.6.77 using rpi-update.
The hash is based on this commit:
kernel: Bump to 6.6.77

sudo rpi-update 8bd0d75837d6a99138c7cbbc24888601b220405a

Restart your Pi to load the downgraded kernel

sudo reboot

Verify the Kernel Version
Once rebooted, confirm the change by checking the kernel version:

uname -r

Now, your Raspberry Pi should be running the older, stable kernel and the examples should work properly again.

The output should now start with 6.6.77.

To revert back to the Main Kernel:

Put the stable kernel files back in place

sudo apt install --reinstall raspberrypi-kernel raspberrypi-bootloader

Now, synchronize all other packages to match that new kernel

sudo apt full-upgrade

With the system now in a consistent state, it’s safe to reboot

sudo reboot

If you want kernel 6.12.20, try the solution in here : (not recommended)

The commit for 6.12.20 does not install using rpi-update, please correct us if we are wrong.

We’ll update everyone once we have a permanent fix. Thanks for your patience while we sort this out!

1 Like

I am getting error for using 6.12.20

mobilestack@raspberrypi:~ $ apt list -a raspberrypi-kernel raspberrypi-bootloader
Listing… Done
raspberrypi-bootloader/stable 1:1.20230405-1 arm64

raspberrypi-bootloader/stable 1:1.20230405-1 armhf

raspberrypi-kernel/stable 1:1.20230405-1 arm64

raspberrypi-kernel/stable 1:1.20230405-1 armhf

mobilestack@raspberrypi:~ $ # For kernel 6.12.20 (adjust version string as needed)
sudo apt install --reinstall
raspberrypi-bootloader=1:6.12.20-1+rpt1
raspberrypi-kernel=1:6.12.20-1+rpt1
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Package raspberrypi-kernel is a virtual package provided by:
raspberrypi-kernel:armhf 1:1.20230405-1 (= 1:1.20230405-1)
You should explicitly select one to install.

Package raspberrypi-bootloader is a virtual package provided by:
raspberrypi-bootloader:armhf 1:1.20230405-1 (= 1:1.20230405-1)
You should explicitly select one to install.

E: Version ‘1:6.12.20-1+rpt1’ for ‘raspberrypi-bootloader’ was not found
E: Version ‘1:6.12.20-1+rpt1’ for ‘raspberrypi-kernel’ was not found
mobilestack@raspberrypi:~ $ sudo apt install --reinstall
raspberrypi-bootloader=1:6.6.74-1+rpt1
raspberrypi-kernel=1:6.6.74-1+rpt1
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Package raspberrypi-kernel is a virtual package provided by:
raspberrypi-kernel:armhf 1:1.20230405-1 (= 1:1.20230405-1)
You should explicitly select one to install.

Package raspberrypi-bootloader is a virtual package provided by:
raspberrypi-bootloader:armhf 1:1.20230405-1 (= 1:1.20230405-1)
You should explicitly select one to install.

Hey @Praveen_Gupta try the steps from the 3rd option mentioned above. You can find the link above.

PCI driver is inconsistent - giving error code (1)

dkms: autoinstall for kernel: 6.12.20+rpt-rpi-2712.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.12.20+rpt-rpi-2712
Setting up linux-headers-6.12.20+rpt-common-rpi (1:6.12.20-1+rpt1~bpo12+1) …
Setting up linux-kbuild-6.12.20+rpt (1:6.12.20-1+rpt1~bpo12+1) …
Setting up hailort-pcie-driver (4.21.0) …

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

build-essential/stable,now 12.9 arm64 [installed]
Do you wish to use DKMS? [Y/n]:
Failed. Exited with status 1. See /var/log/hailort-pcie-driver.deb.log
dpkg: error processing package hailort-pcie-driver (–configure):
installed hailort-pcie-driver package post-installation script subprocess retur
ned error exit status 1
Setting up linux-headers-6.12.20+rpt-rpi-2712 (1:6.12.20-1+rpt1~bpo12+1) …
/etc/kernel/header_postinst.d/dkms:
dkms: running auto installation service for kernel 6.12.20+rpt-rpi-2712.
Sign command: /lib/modules/6.12.20+rpt-rpi-2712/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module:
Cleaning build area…
make -j4 KERNELRELEASE=6.12.20+rpt-rpi-2712 -C /lib/modules/6.12.20+rpt-rpi-2712
/build M=/var/lib/dkms/hailo_pci/4.21.0/build/linux/pcie modules…
Signing module /var/lib/dkms/hailo_pci/4.21.0/build/./linux/pcie/hailo_pci.ko
Cleaning build area…
Forcing version override of hailo_pci

hailo_pci.ko.xz:
Running module version sanity check.

  • Original module
  • Installation
    • Installing to /lib/modules/6.12.20+rpt-rpi-2712/updates/dkms/
      depmod…
      dkms: autoinstall for kernel: 6.12.20+rpt-rpi-2712.
      Errors were encountered while processing:
      hailort-pcie-driver
      E: Sub-process /usr/bin/dpkg returned an error code (1)
      mobilestack@raspberrypi:~/hailo-rpi5-examples $

Hey @Praveen_Gupta,

Welcome to the Hailo Community!

It looks like you’re using Driver 4.21, which hasn’t been officially released for Raspberry Pi yet. We recommend sticking with the officially supported version: hailo-all, provided for Raspberry Pi compatibility.

Yes. But, then, there are instruction to perform sudo apt update; sudo apt upgrade

and this cause problems.

Looks like, different instructions for rp5-examples and Frigate installations is making the whole effort inconsistent. Again, I am not trying to be negative. Just, identifying issues that newbie face as they try RP5 with Hailo8L AI HAT.

RP5 example installation work upto “python basic_pipelines/detection.py” and then nothing else for segmentation / Pose detection works.

As one do Frigate installation after RP5 (upto detection), one gets into authentication issue and there are no clean instructions of how to overcome authentication and use RP5 Camera Module-3 with Frigate.

Thanks
-Praveen

FWIW, I managed to fix this by running rpi-update

Hi, I jave the 8Gig version with the AI Hat+ (26Tops) version and Linux 6.12.25 firmware

When I ran the tests for the first time, I found over 9 or more failing tests (number of failed tests vary on each try), after searching I found this post to downgrade version, I couldn’t downgrade to 6.6.74 but was able to downgrade to 6.12.20.

After downgrading there is no more failed tests being displayed as result of the script, but dmesg tells a completely different story, look, before downgrading it when the tests of the script failed the errors were logged by dmesg as:

[62355.282370] hailo: get_user_pages failed with -4
[62355.282382] hailo 0001:01:00.0: failed to set sg list for user buffer -4
[62355.282387] hailo 0001:01:00.0: failed map buffer 7ffe4bd80000

But now, besides no failed tests in the result, dmesg it is flooded with stacktraces:

[ 1247.301098] ------------[ cut here ]------------
[ 1247.301099] WARNING: CPU: 2 PID: 16922 at include/linux/rwsem.h:80 find_vma+0x64/0x78
[ 1247.301103] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep binfmt_misc ov5647 brcmfmac_wcc spidev uvcvideo hci_uart videobuf2_vmalloc uvc btbcm bluetooth brcmfmac aes_ce_blk aes_ce_cipher ghash_ce rp1_cfe gf128mul rpi_hevc_dec sha2_ce pisp_be v4l2_mem2mem v4l2_fwnode ecdh_generic sha256_arm64 v4l2_async videobuf2_dma_contig sha1_ce ecc brcmutil videobuf2_memops videobuf2_v4l2 videodev cfg80211 hailo_pci libaes sha1_generic rfkill raspberrypi_hwmon videobuf2_common spi_bcm2835 mc i2c_brcmstb rp1_pio gpio_keys rp1 raspberrypi_gpiomem i2c_designware_platform rp1_adc pwm_fan i2c_designware_core rp1_mailbox nvmem_rmem i2c_dev ledtrig_pattern fuse dm_mod ip_tables x_tables ipv6 vc4 snd_soc_hdmi_codec snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd drm_display_helper v3d drm_dma_helper cec drm_shmem_helper drm_kms_helper gpu_sched drm drm_panel_orientation_quirks backlight uio_pdrv_genirq uio
[ 1247.301161] CPU: 2 UID: 1000 PID: 16922 Comm: inference_hailo Tainted: G W 6.12.20+rpt-rpi-2712 #1 Debian 1:6.12.20-1+rpt1~bpo12+1
[ 1247.301164] Tainted: [W]=WARN
[ 1247.301165] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[ 1247.301166] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=–)
[ 1247.301169] pc : find_vma+0x64/0x78
[ 1247.301172] lr : hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301176] sp : ffffc0008b5f3c20
[ 1247.301177] x29: ffffc0008b5f3c20 x28: ffff80000e80c600 x27: 0000000000000000
[ 1247.301180] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 1247.301184] x23: 0000000000000002 x22: ffff800140e1f0c8 x21: 0000000000009600
[ 1247.301187] x20: 00007fffa2b24000 x19: ffff800140007780 x18: 0000000000000000
[ 1247.301190] x17: 0000000000000000 x16: ffffd06fce72d528 x15: 00007fff60f9cff0
[ 1247.301193] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1247.301196] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fc56e5634
[ 1247.301200] x8 : ffff800140007800 x7 : 0000000000000000 x6 : 000000000000003f
[ 1247.301203] x5 : 0000000000000040 x4 : 0000000000000080 x3 : 0000000000000000
[ 1247.301206] x2 : ffff80000e80c600 x1 : 0000000000000000 x0 : ffff800100be1c80
[ 1247.301209] Call trace:
[ 1247.301210] find_vma+0x64/0x78
[ 1247.301213] hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301218] hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
[ 1247.301221] hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
[ 1247.301225] hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
[ 1247.301229] __arm64_sys_ioctl+0xb4/0x100
[ 1247.301231] invoke_syscall+0x50/0x120
[ 1247.301235] el0_svc_common.constprop.0+0x48/0xf0
[ 1247.301238] do_el0_svc+0x24/0x38
[ 1247.301241] el0_svc+0x30/0xd0
[ 1247.301244] el0t_64_sync_handler+0x100/0x130
[ 1247.301248] el0t_64_sync+0x190/0x198
[ 1247.301250] —[ end trace 0000000000000000 ]—
[ 1247.301278] ------------[ cut here ]------------
[ 1247.301280] WARNING: CPU: 2 PID: 16922 at include/linux/rwsem.h:80 find_vma+0x64/0x78
[ 1247.301284] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep binfmt_misc ov5647 brcmfmac_wcc spidev uvcvideo hci_uart videobuf2_vmalloc uvc btbcm bluetooth brcmfmac aes_ce_blk aes_ce_cipher ghash_ce rp1_cfe gf128mul rpi_hevc_dec sha2_ce pisp_be v4l2_mem2mem v4l2_fwnode ecdh_generic sha256_arm64 v4l2_async videobuf2_dma_contig sha1_ce ecc brcmutil videobuf2_memops videobuf2_v4l2 videodev cfg80211 hailo_pci libaes sha1_generic rfkill raspberrypi_hwmon videobuf2_common spi_bcm2835 mc i2c_brcmstb rp1_pio gpio_keys rp1 raspberrypi_gpiomem i2c_designware_platform rp1_adc pwm_fan i2c_designware_core rp1_mailbox nvmem_rmem i2c_dev ledtrig_pattern fuse dm_mod ip_tables x_tables ipv6 vc4 snd_soc_hdmi_codec snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd drm_display_helper v3d drm_dma_helper cec drm_shmem_helper drm_kms_helper gpu_sched drm drm_panel_orientation_quirks backlight uio_pdrv_genirq uio
[ 1247.301342] CPU: 2 UID: 1000 PID: 16922 Comm: inference_hailo Tainted: G W 6.12.20+rpt-rpi-2712 #1 Debian 1:6.12.20-1+rpt1~bpo12+1
[ 1247.301345] Tainted: [W]=WARN
[ 1247.301346] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[ 1247.301347] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=–)
[ 1247.301350] pc : find_vma+0x64/0x78
[ 1247.301353] lr : hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301357] sp : ffffc0008b5f3c20
[ 1247.301358] x29: ffffc0008b5f3c20 x28: ffff80000e80c600 x27: 0000000000000000
[ 1247.301361] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 1247.301365] x23: 0000000000000002 x22: ffff800140e1f0c8 x21: 0000000000009600
[ 1247.301368] x20: 00007fffa2af4000 x19: ffff800140006700 x18: 0000000000000000
[ 1247.301371] x17: 0000000000000000 x16: ffffd06fce72d528 x15: 00007fff60f9cff0
[ 1247.301375] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1247.301378] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fc56e5634
[ 1247.301381] x8 : ffff800140006780 x7 : 0000000000000000 x6 : 000000000000003f
[ 1247.301384] x5 : 0000000000000040 x4 : 0000000000000080 x3 : 0000000000000000
[ 1247.301387] x2 : ffff80000e80c600 x1 : 0000000000000000 x0 : ffff800100be1c80
[ 1247.301390] Call trace:
[ 1247.301391] find_vma+0x64/0x78
[ 1247.301394] hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301398] hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
[ 1247.301402] hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
[ 1247.301406] hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
[ 1247.301409] __arm64_sys_ioctl+0xb4/0x100
[ 1247.301412] invoke_syscall+0x50/0x120
[ 1247.301415] el0_svc_common.constprop.0+0x48/0xf0
[ 1247.301419] do_el0_svc+0x24/0x38
[ 1247.301422] el0_svc+0x30/0xd0
[ 1247.301425] el0t_64_sync_handler+0x100/0x130
[ 1247.301428] el0t_64_sync+0x190/0x198
[ 1247.301430] —[ end trace 0000000000000000 ]—
[ 1247.301459] ------------[ cut here ]------------
[ 1247.301460] WARNING: CPU: 2 PID: 16922 at include/linux/rwsem.h:80 find_vma+0x64/0x78
[ 1247.301464] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep binfmt_misc ov5647 brcmfmac_wcc spidev uvcvideo hci_uart videobuf2_vmalloc uvc btbcm bluetooth brcmfmac aes_ce_blk aes_ce_cipher ghash_ce rp1_cfe gf128mul rpi_hevc_dec sha2_ce pisp_be v4l2_mem2mem v4l2_fwnode ecdh_generic sha256_arm64 v4l2_async videobuf2_dma_contig sha1_ce ecc brcmutil videobuf2_memops videobuf2_v4l2 videodev cfg80211 hailo_pci libaes sha1_generic rfkill raspberrypi_hwmon videobuf2_common spi_bcm2835 mc i2c_brcmstb rp1_pio gpio_keys rp1 raspberrypi_gpiomem i2c_designware_platform rp1_adc pwm_fan i2c_designware_core rp1_mailbox nvmem_rmem i2c_dev ledtrig_pattern fuse dm_mod ip_tables x_tables ipv6 vc4 snd_soc_hdmi_codec snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd drm_display_helper v3d drm_dma_helper cec drm_shmem_helper drm_kms_helper gpu_sched drm drm_panel_orientation_quirks backlight uio_pdrv_genirq uio
[ 1247.301522] CPU: 2 UID: 1000 PID: 16922 Comm: inference_hailo Tainted: G W 6.12.20+rpt-rpi-2712 #1 Debian 1:6.12.20-1+rpt1~bpo12+1
[ 1247.301524] Tainted: [W]=WARN
[ 1247.301526] Hardware name: Raspberry Pi 5 Model B Rev 1.0 (DT)
[ 1247.301527] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=–)
[ 1247.301529] pc : find_vma+0x64/0x78
[ 1247.301532] lr : hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301536] sp : ffffc0008b5f3c20
[ 1247.301537] x29: ffffc0008b5f3c20 x28: ffff80000e80c600 x27: 0000000000000000
[ 1247.301540] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 1247.301544] x23: 0000000000000002 x22: ffff800140e1f0c8 x21: 0000000000009600
[ 1247.301547] x20: 00007fffa2ac4000 x19: ffff800140006d80 x18: 0000000000000000
[ 1247.301550] x17: 0000000000000000 x16: ffffd06fce72d528 x15: 00007fff60f9cff0
[ 1247.301553] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 1247.301556] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffd06fc56e5634
[ 1247.301559] x8 : ffff800140006e00 x7 : 0000000000000000 x6 : 000000000000003f
[ 1247.301563] x5 : 0000000000000040 x4 : 0000000000000080 x3 : 0000000000000000
[ 1247.301566] x2 : ffff80000e80c600 x1 : 0000000000000000 x0 : ffff800100be1c80
[ 1247.301569] Call trace:
[ 1247.301570] find_vma+0x64/0x78
[ 1247.301573] hailo_vdma_buffer_map+0x13c/0x610 [hailo_pci]
[ 1247.301577] hailo_vdma_buffer_map_ioctl+0xd8/0x350 [hailo_pci]
[ 1247.301581] hailo_vdma_ioctl+0x1d4/0x268 [hailo_pci]
[ 1247.301584] hailo_pcie_fops_unlockedioctl+0x16c/0x7c8 [hailo_pci]
[ 1247.301588] __arm64_sys_ioctl+0xb4/0x100
[ 1247.301591] invoke_syscall+0x50/0x120
[ 1247.301594] el0_svc_common.constprop.0+0x48/0xf0
[ 1247.301597] do_el0_svc+0x24/0x38
[ 1247.301600] el0_svc+0x30/0xd0
[ 1247.301603] el0t_64_sync_handler+0x100/0x130
[ 1247.301606] el0t_64_sync+0x190/0x198
[ 1247.301608] —[ end trace 0000000000000000 ]—

Hi,

I think a warning should be added in the github installation notes.

I have worked one month with a not stable environment and now it’s finally stable downgrading the kernel.

Ok, I found in the official raspberri-pi forum the actual fix to the problem

Original post

sudo rpi-update 8bd0d75837d6a99138c7cbbc24888601b220405a

After this command the kernel version is “6.6.77-v8-16k+”, it passes all 37 tests with only a single error shown in dmesg:

[ 104.317985] source of link ‘rp1-cfe-fe_config’:0->‘pisp-fe’:1 is not a V4L2 sub-device, driver bug!
[ 104.318005] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 204.035820] hailo 0000:01:00.0: unlockedioctl down_interruptible failed
[ 225.264365] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 245.472467] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 265.788532] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 286.148692] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 306.589480] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 1694.514122] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 1704.581470] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 1714.569465] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 1724.612285] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps
[ 1734.669570] rp1-cfe 1f00110000.csi: Using a link rate of 437 Mbps

Hey All ,

If you are still facing this issue please follow 🎉 Kernel Bug Fix - Update Available! - #2