Fw-control error

Hello,

we have below error message while we installing and testing hailo-8.
Please advise to me.

  • CPU : Qualcomm RB5 platform
  • NPU : Hailo-8 x 1
  • OS : Ubuntu 20.04 kernel 5.4.xx

Q1. After install driver and RT,
we can check hailortcli scan and hailortcli -v
but error happens if we run ‘hailortcli fw-control identify’.


Please advise against this error.

Q2. we can use below command?
we don;t use -x when tesing above fw-control error issue.
thsi is just question for our internal test.

dpkg -x hailort-pcie-driver_4.17.0_all.deb hailort-pcie-driver_4.17.0_all

-x is necessary for customer’s system because of system’s internal security policy. (According to customer, ko file can’t be used in system.)

thanks

add more history…

Based on current history, we tried like below and error message.
thanks

root@Airpath:~/test# hailortcli run yolov8s.hef
Running streaming inference (yolov8s.hef):
Transform data: true
Type: auto
Quantized: true
[HailoRT] [error] CHECK failed - Failed to open device file /dev/hailo0 with error 6
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36) - Failed creating vdevice

Hey @jy.han

can you run hailortcli scan to check if the device is recognized .
you can also check out
hailo-rpi5-examples/doc/install-raspberry-pi5.md at main · hailo-ai/hailo-rpi5-examples · GitHub

Regards

@omria
as i reported, we tried “hailortcli scan and -v” and ok.
we also tried lspci and found hailo device correctly.

Ok , can you please run :

sudo dpkg --purge hailort-pcie-driver

and re-install the driver .
I am going to check out the error 6 and will get back to you.

Regards

Hey @jy.han

Please update the kernel version ( Ubuntu OS) to the latest , this has solved the problem for our other costumers.

Hope this fixes the issue
Regards

I will ask this…

one more, DKMS must be installed?
If DKMS not installed??

When press ‘Y’ at the step of DKMS installation, DPKG erroe happen.
so tester pressed ‘N’, and driver installed.

possible?

the DKMS isn’t a must to be installed.
The driver can be installed and will work without the DKMS

Regards

ok, very thanks for your answer !!

1 Like

Hello,

my partner have same error log after lots of trial.
we tried previous driver version and same results.

Except trial with the latest kernel source, any idea to solve this issue??

thanks

Is happening always? If you bring up the system afresh, is it ok?
If not, have causing it to get into this state?

still under debugging.
but not any idea now.
partner trying to update the kernel source, but very difficult because this hw and os image depends on Qualcomm.
and we’re not sure that we can solve issue with kernel update.
very strange status.

What about what I’ve asked?

sorry, didn’t understand your intension.
problem always happens.
we tried several pcie driver and changed several hardwares.
but still same error when testing hailortcli fw-control.

What’s the output from dmesg?

@Nadav Hello,
Below is log related to Hailo-8 from dmesg.
I think Hailo driver was loaded correctly.

I can send whole log file to you if you let me know e-mail thru chat.
(Whole log can’t be opened in this community.)

< log >
[ 13.571262] hailo: Init module. driver version 4.17.0
[ 13.580058] hailo 0002:01:00.0: Adding to iommu group 48
[ 13.582246] hailo 0002:01:00.0: Probing on: 1e60:2864…
[ 13.582248] hailo 0002:01:00.0: Probing: Allocate memory for device extension, 11600
[ 13.582265] pci-msm-rc 0002:00:00.0: enabling device (0006 → 0007)
[ 13.582337] hailo 0002:01:00.0: enabling device (0000 → 0002)
[ 13.582347] hailo 0002:01:00.0: Probing: Device enabled
[ 13.582413] hailo 0002:01:00.0: Probing: mapped bar 0 - 00000000f03972c9 16384
[ 13.582418] hailo 0002:01:00.0: Probing: mapped bar 2 - 0000000039b43d55 4096
[ 13.582421] hailo 0002:01:00.0: Probing: mapped bar 4 - 00000000c767472b 16384
[ 13.582426] hailo 0002:01:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[ 13.582456] hailo 0002:01:00.0: Probing: Enabled 64 bit dma
[ 13.582462] hailo 0002:01:00.0: Probing: Using specialized dma_ops=fast_smmu_dma_ops
[ 13.582463] hailo 0002:01:00.0: Probing: Using userspace allocated vdma buffers
[ 13.582469] hailo 0002:01:00.0: Disabling ASPM L0s
[ 13.582473] hailo 0002:01:00.0: Successfully disabled ASPM L0s
[ 13.678622] hailo 0002:01:00.0: Firmware was loaded successfully
[ 13.692810] hailo 0002:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0

thanks

@Nadav and i also want to know about error code of errno:110 and error 6.

yes, we can’t guess everthing from error code but i think we need to find something from error code because we didn’t find any idea for long times.

thanks

@Nadav
Below is dmesg log from another ARM CPU(RK3588) based hardware.
we installed hailo driver and RT on this board and also working fw-control well…

just for comparison with our issued HW(Qualcomm RB5)…
I think issued board also have hailo driver correctly…
log was very similar with RK3588…
(FYI)

< DMESG Log captured from RK3588 based boards >

/Downloads/hailort-drivers/linux/pcie$ dmesg | tail
[ 3838.811371] hailo 0003:31:00.0: Probing: mapped bar 0 - 00000000fa539104 16384
[ 3838.811377] hailo 0003:31:00.0: Probing: mapped bar 2 - 0000000058a8e023 4096
[ 3838.811381] hailo 0003:31:00.0: Probing: mapped bar 4 - 00000000693c06ef 16384
[ 3838.811386] hailo 0003:31:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[ 3838.811394] hailo 0003:31:00.0: Probing: Enabled 64 bit dma
[ 3838.811396] hailo 0003:31:00.0: Probing: Using userspace allocated vdma buffers
[ 3838.811401] hailo 0003:31:00.0: Disabling ASPM L0s
[ 3838.811406] hailo 0003:31:00.0: Successfully disabled ASPM L0s
[ 3838.987759] hailo 0003:31:00.0: Firmware was loaded successfully
[ 3838.999041] hailo 0003:31:00.0: Probing: Added board 1e60-2864, /dev/hailo0

any feedback about my update,pls ?

And i want to know the error code, errno 110.

thanks

Hi Jy,
There’s no simple answer on this, Qualcomm doen’t open source their Linux, so we cannot test it on our side. There are a few leads you can take:
** HW

  • make sure that the socket is for PCIe and not SATA.
  • delivers enough power

** SW

  • Need to debug the driver, add see if the MSI interrupts are coming through, if the FW is loaded correctly.