I am trying to test the models I train by GPU on Ubuntu 22.04 and compiled by DFC to test run on a hailo8l installed on a pci card in my pc. (later I will use RPi5 for inference)
the hailo8l is the same one as the ones offered for RPi5 and I can see that the motherboard recognizes it as below:
06:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
when I try to install the driver, I get below error, tried almost everything I found in this community and in github, with no success, the log is also copied further down.
Error:
(Reading database … 197616 files and directories currently installed.)
Preparing to unpack hailort-pcie-driver_4.18.0_all.deb …
Verifying secureboot is disabled.
Unpacking hailort-pcie-driver (4.18.0) over (4.18.0) …
Setting up hailort-pcie-driver (4.18.0) …
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
build-essential/jammy,now 12.9ubuntu3 amd64 [installed]
Do you wish to use DKMS? [Y/n]:
Failed to install PCIe driver to the DKMS tree. Trying to install PCIe driver without DKMS
Failed. Exited with status 2. See /var/log/hailort-pcie-driver.deb.log
dpkg: error processing package hailort-pcie-driver (–install):
installed hailort-pcie-driver package post-installation script subprocess returned error exit status 2
Errors were encountered while processing:
hailort-pcie-driver
Log:
cc1: all warnings being treated as errors
make[3]: *** [scripts/Makefile.build:243: /var/lib/dkms/hailo_pci/4.18.0/build/linux/pcie/…/…/common/pcie_common.o] Error 1
make[3]: *** Waiting for unfinished jobs…
make[2]: *** [/usr/src/linux-headers-6.8.0-40-generic/Makefile:1926: /var/lib/dkms/hailo_pci/4.18.0/build/linux/pcie] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make[1]: Leaving directory ‘/usr/src/linux-headers-6.8.0-40-generic’
make: *** [Makefile:131: install_dkms] Error 1
make[1]: Entering directory ‘/usr/src/linux-headers-6.8.0-40-generic’
make[1]: Leaving directory ‘/usr/src/linux-headers-6.8.0-40-generic’
make[1]: Entering directory ‘/usr/src/linux-headers-6.8.0-40-generic’
warning: the compiler differs from the one used to build the kernel
The kernel was built by: x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
You are using: gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
CC [M] /share/opt/hailo/linux/pcie/src/pcie.o
CC [M] /share/opt/hailo/linux/pcie/src/fops.o
CC [M] /share/opt/hailo/linux/pcie/src/utils.o
CC [M] /share/opt/hailo/linux/pcie/src/sysfs.o
CC [M] /share/opt/hailo/linux/pcie/src/pci_soc_ioctl.o
CC [M] /share/opt/hailo/linux/pcie/…/…/common/fw_validation.o
CC [M] /share/opt/hailo/linux/pcie/…/…/common/fw_operation.o
CC [M] /share/opt/hailo/linux/pcie/…/…/common/pcie_common.o
/share/opt/hailo/linux/pcie/…/…/common/pcie_common.c:659:6: error: no previous prototype for ‘hailo_pcie_is_device_ready_for_boot’ [-Werror=missing-prototypes]
659 | bool hailo_pcie_is_device_ready_for_boot(struct hailo_pcie_resources *resources)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[3]: *** [scripts/Makefile.build:243: /share/opt/hailo/linux/pcie/…/…/common/pcie_common.o] Error 1
make[2]: *** [/usr/src/linux-headers-6.8.0-40-generic/Makefile:1926: /share/opt/hailo/linux/pcie] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make[1]: Leaving directory ‘/usr/src/linux-headers-6.8.0-40-generic’
make: *** [Makefile:93: all] Error 2
Failed. Exited with status 2. See /var/log/hailort-pcie-driver.deb.log
Failed at 49
46
47 function compile_and_install_pcie_driver_without_dkms() {
48 make clean &>> $LOG
49 make all &>> $LOG
50 make install &>> $LOG
51 }
52
I have purged the half installed driver, cleanup all leftovers, re-installed a fresh one with errors showing again, restarted the pc and checked with “HailoRT hailortcli scan”. below is the result of both attempts.
(when I download the driver the size of the file is 136kb)
Your help is much appreciated.
I have no clue other than it says in the log that warnings are treated as errors.
Selecting previously unselected package hailort-pcie-driver.
dpkg: warning: files list file for package ‘hailort’ missing; assuming package has no files currently installed
(Reading database … 198792 files and directories currently installed.)
Preparing to unpack hailort-pcie-driver_4.18.0_all.deb …
Verifying secureboot is disabled.
Unpacking hailort-pcie-driver (4.18.0) …
Setting up hailort-pcie-driver (4.18.0) …
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
build-essential/jammy,now 12.9ubuntu3 amd64 [installed]
Do you wish to use DKMS? [Y/n]:
Failed to install PCIe driver to the DKMS tree. Trying to install PCIe driver without DKMS
Failed. Exited with status 2. See /var/log/hailort-pcie-driver.deb.log
dpkg: error processing package hailort-pcie-driver (–install):
installed hailort-pcie-driver package post-installation script subprocess returned error exit status 2
Errors were encountered while processing:
hailort-pcie-driver
HailoRT scan:
[HailoRT] [error] Can’t find hailo pcie class, this may happen if the driver is not installed (this may happen if the kernel was updated), or if there is no connected Hailo board
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64) - Failed listing pcie devices
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_PCIE_DRIVER_NOT_INSTALLED(64)
The issue is kernel compatibility. You’re using kernel 6.8, but the Hailo PCIe driver only supports 6.5. Downgrade to kernel 6.5 and reinstall the driver to fix it. Let me know if you need help with the downgrade or re-installation!
thanks for the help, this worked. that made me happy.
(now I have the GPU driver not working on the 6.5 kernel but that task is on me.)
just checking with you to make sure everything is in order, below is the outcome of the hailortcli scan, is this correct representation of found hailo?
Hailo Devices:
[-] Device: 0000:06:00.0
and “hailortcli fw-control identify” shows the details of the device.
I am asking because when I run inference using the tutorial I get below error:
HailoRTException: libhailort failed with error: 74 (HAILO_OUT_OF_PHYSICAL_DEVICES)
The error you’re seeing is because the previous app didn’t release the Hailo device properly. You’ll need to stop the app to ensure the device is released.
thanks again, Ive been busy finding one kernel that is compatible drivers for both RTX4070S GPU and hailo unit, with no avail. the RTX4070S is quite new and maybe that is why the drivers dont work with older kernels. I probably have to wait maybe hailo team upgrades the hailo driver to be compatible with 6.8.
regarding last note, I checked and there were no other app working with the device. I will reinstall and see what happens.