Hailo8 dmesg errors with new kernel

Hello,

recently we updated kernel from

Ubuntu 24.04.2 LTS, Linux test 6.8.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 2 20:41:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

to

Ubuntu 24.04.3 LTS, Linux TA120-C001 6.14.0-29-generic #29~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 14 16:52:50 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

we started to get warnings in dmesg.

I tried to build drivers from source, but that did not help.

We are using drivers version 4.20.0. but same thing happend with 4.22.0 as well.

We are going to need newer kernel for CPU driver compatibility.

I could not attach full dmesg log, so here is only the last bit, full log had ~2,5k lines, I think the error produced by our test was still the same only repeated many times anyway.

How do we fix this please? Have anyone run into this problem?

Thank you for any suggestion,

Daniel

[  514.793444] ------------[ cut here ]------------
[  514.793446] WARNING: CPU: 0 PID: 2245 at include/linux/rwsem.h:80 find_vma+0x5b/0x70
[  514.793453] Modules linked in: tcp_diag raw_diag inet_diag cfg80211 sch_fq_codel xe drm_gpuvm gpu_sched drm_ttm_helper msr drm_exec cdc_acm drm_suballoc_helper snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda_mlink snd_sof_intel_hda snd_hda_codec_hdmi snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_acpi_intel_match snd_soc_acpi_intel_sdca_quirks intel_rapl_msr soundwire_generic_allocation snd_soc_acpi intel_rapl_common soundwire_bus snd_soc_sdca snd_soc_avs snd_soc_hda_codec snd_hda_ext_core x86_pkg_temp_thermal intel_powerclamp snd_soc_core coretemp snd_compress ac97_bus cmdlinepart snd_pcm_dmaengine kvm_intel spi_nor snd_usb_audio snd_hda_intel snd_intel_dspcfg snd_usbmidi_lib snd_intel_sdw_acpi mtd mei_hdcp mei_pxp snd_hda_codec kvm snd_ump snd_rawmidi snd_hda_core snd_hwdep snd_seq_device snd_pcm irqbypass i2c_i801 rapl snd_timer wmi_bmof intel_cstate hailo_pci(OE) i2c_smbus mei_me snd
[  514.793522]  spi_intel_pci i2c_mux soundcore spi_intel mei mc intel_pmc_core igen6_edac nls_iso8859_1 pmt_telemetry pmt_class intel_vsec acpi_pad acpi_tad upboard_fpga input_leds mac_hid binfmt_misc i915 dm_multipath drm_buddy ttm drm_display_helper cec rc_core i2c_algo_bit efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 linear uas usb_storage hid_cmedia spi_pxa2xx_platform dw_dmac dw_dmac_core 8250_dw spi_pxa2xx_core mmc_block rpmb_core sdhci_pci polyval_clmulni sdhci_uhs2 polyval_generic ghash_clmulni_intel r8169 sdhci sha256_ssse3 intel_ish_ipc sha1_ssse3 cqhci intel_lpss_pci realtek intel_ishtp intel_lpss idma64 video wmi pinctrl_alderlake hid_generic usbhid hid aesni_intel crypto_simd cryptd
[  514.793596] CPU: 0 UID: 0 PID: 2245 Comm: det Tainted: G        W  OE      6.14.0-29-generic #29~24.04.1-Ubuntu
[  514.793600] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[  514.793602] Hardware name: AAEON UPS-ASL01/UPS-ASL01, BIOS USASAM12 08/14/2024
[  514.793604] RIP: 0010:find_vma+0x5b/0x70
[  514.793609] Code: c7 c2 ff ff ff ff 48 8d 75 f0 e8 10 ac db 00 48 8b 55 f8 65 48 2b 14 25 28 00 00 00 75 10 c9 31 d2 31 f6 31 ff c3 cc cc cc cc <0f> 0b eb cd e8 8c 3f de 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90
[  514.793612] RSP: 0018:ffff9de1c3eb7ae0 EFLAGS: 00010246
[  514.793614] RAX: 0000000000000000 RBX: ffff8e49da20f600 RCX: 0000000000000000
[  514.793616] RDX: 0000000000000000 RSI: 00007a45b6830000 RDI: ffff8e49c7a6bc80
[  514.793619] RBP: ffff9de1c3eb7af0 R08: 0000000000000000 R09: 0000000000000000
[  514.793620] R10: 0000000000000000 R11: 0000000000000000 R12: 00007a45b6830000
[  514.793623] R13: ffff8e49c1d8b0c8 R14: 0000000000000000 R15: 0000000000000000
[  514.793625] FS:  00007a45bb7b8740(0000) GS:ffff8e4b37a00000(0000) knlGS:0000000000000000
[  514.793627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  514.793629] CR2: 00007a45b6833000 CR3: 000000011c9de001 CR4: 0000000000f72ef0
[  514.793632] PKRU: 55555554
[  514.793633] Call Trace:
[  514.793635]  <TASK>
[  514.793637]  hailo_vdma_buffer_map+0x1a6/0x860 [hailo_pci]
[  514.793649]  hailo_vdma_buffer_map_ioctl+0xb4/0x260 [hailo_pci]
[  514.793659]  hailo_vdma_ioctl+0x258/0x360 [hailo_pci]
[  514.793670]  hailo_pcie_fops_unlockedioctl+0x18b/0x6f0 [hailo_pci]
[  514.793680]  __x64_sys_ioctl+0xa4/0xe0
[  514.793686]  x64_sys_call+0x131e/0x2650
[  514.793691]  do_syscall_64+0x7e/0x170
[  514.793695]  ? arch_exit_to_user_mode_prepare.isra.0+0x22/0xd0
[  514.793700]  ? syscall_exit_to_user_mode+0x38/0x1d0
[  514.793704]  ? do_syscall_64+0x8a/0x170
[  514.793707]  ? _raw_spin_lock_irqsave+0xe/0x20
[  514.793711]  ? up+0x58/0xa0
[  514.793716]  ? hailo_pcie_fops_unlockedioctl+0x26d/0x6f0 [hailo_pci]
[  514.793725]  ? __handle_mm_fault+0x62c/0x770
[  514.793729]  ? arch_exit_to_user_mode_prepare.isra.0+0x22/0xd0
[  514.793734]  ? syscall_exit_to_user_mode+0x38/0x1d0
[  514.793738]  ? do_syscall_64+0x8a/0x170
[  514.793741]  ? count_memcg_events.constprop.0+0x2a/0x50
[  514.793745]  ? handle_mm_fault+0x1df/0x2d0
[  514.793749]  ? do_user_addr_fault+0x1d2/0x870
[  514.793754]  ? arch_exit_to_user_mode_prepare.isra.0+0x22/0xd0
[  514.793759]  ? irqentry_exit_to_user_mode+0x2d/0x1d0
[  514.793763]  ? irqentry_exit+0x43/0x50
[  514.793767]  ? exc_page_fault+0x96/0x1e0
[  514.793771]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  514.793776] RIP: 0033:0x7a45b9f24ded
[  514.793779] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
[  514.793781] RSP: 002b:00007ffc0f2123f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  514.793784] RAX: ffffffffffffffda RBX: 00007ffc0f212700 RCX: 00007a45b9f24ded
[  514.793787] RDX: 00007ffc0f2124a0 RSI: 00000000c0287604 RDI: 0000000000000003
[  514.793789] RBP: 00007ffc0f212440 R08: 0000000000000000 R09: 0000000000000000
[  514.793791] R10: 0000561cfd292360 R11: 0000000000000246 R12: 0000561cfd292310
[  514.793793] R13: 0000000000000000 R14: 0000561cfd292280 R15: 0000561cfd292338
[  514.793797]  </TASK>
[  514.793798] ---[ end trace 0000000000000000 ]---

Welcome to the Hailo Community!

We usually add new kernel support with every release of our software. The next version should be released in October.

That HailoRT version is older than Linux kernel 6.14. So, our developers could not have validated that kernel version.

Thank you for your response. That should potentially solve our issue.

Can you confirm, that the new version will be compatible with kernel 6.14 or at least >=6.12 and hailo8, please?

Best regards,

Daniel

Looks like my colleagues released HailoRT 4.23 already. You can download it from the Developer Zone. GitHub has not been updated yet.

1 Like

We tested it on two machines, but with same result.

Is there anything else we can do to fix this?