Hailo-8 c-processor not detected by hailortcli etc when on a geekworm PCIe switch for RPI5

General
,
1

Hello!

I’ve been running the RPI AI devkit with great success. I’ve used it successfully with the original RPi M.2 hat that came with the kit, and on the Pimoroni NVMe Base Duo PCIe switch, alongside an NVMe SSD.

However, when I remove the Pimorini NVMe Base Duo and replace it with the Geekworm X1011 PCIe to Four M.2 NVMe SSD Board for Raspberry Pi 5, the Hailo-8 co-processor no longer gets recognized by the Hailo software.

Here’s my environment and a demonstration of the problem:

pi@aye:~ $ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
pi@aye:~ $ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
pi@aye:~ $ uname -r
6.6.47+rpt-rpi-2712
pi@aye:~ $ lsmod | grep hailo
hailo_pci              98304  0
pi@aye:~ $ hailortcli -v
HailoRT-CLI version 4.18.0
pi@aye:~ $ sudo lspci | grep Hailo
0000:06:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
pi@aye:~ $ hailortcli fw-control identify
pi@aye:~ $ hailortcli scan
Hailo devices not found
pi@aye:~ $ hailortcli fw-control identify --bdf 0000:06:00.0
[HailoRT] [error] Requested device not found
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_ARGUMENT(2)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_ARGUMENT(2)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_ARGUMENT(2)
pi@aye:~ $

As you can see, the co-processor is present in the output of lspci, but is not recognized by the hailortctrl command, even when I provide the PCIe address.

When I move the co-processor and NVMe back to the Pimoroni, everything works as expected:

pi@aye:~ $ lsmod | grep hailo
hailo_pci              98304  0
pi@aye:~ $ sudo lspci | grep Hailo
0000:03:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
pi@aye:~ $ hailortcli fw-control identify
Executing on device: 0000:03:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.18.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8L
Serial Number: HLDDLBB242500870
Part Number: HM21LB1C2LAE
Product Name: HAILO-8L AI ACC M.2 B+M KEY MODULE EXT TMP

pi@aye:~ $ hailortcli scan
Hailo Devices:
[-] Device: 0000:03:00.0
pi@aye:~ $

The only difference in configuration is that on the Geekworm PCIe switch, the Hailo co-processor enumerates at location 06:00.0, while on the Pimoroni PCIe switch, it enumerates at location 03:00.0.

So I realize this is most likely due to a deficiency in the Geekworm PCIe switch, but I’m hoping that perhaps there’s another workaround I can try. Thank you for your time!

Jeremy

2

Hey @jeremy.impson

Welcome to the Hailo Community!

I understand that the device is detected by lspci but not recognized by hailortcli. Let’s go through some troubleshooting steps:

  1. Check PCIe Link:
    Run sudo lspci -vvv -s 06:00.0 and compare the output with your working Pimoroni setup. Look for differences in link speed or width.

  2. Examine Kernel Logs:
    Use dmesg | grep -i pcie to check for any PCIe-related errors or warnings.

  3. Test Power Supply:
    Try using the Hailo-8 alone on the switch to rule out power issues.

  4. Force Device Identification:
    Run hailortcli fw-control identify --bdf 0000:06:00.0 to attempt direct identification.

  5. Update HailoRT Driver:
    Ensure your HailoRT driver (v4.18.0) is compatible with your setup. Consider rebuilding if necessary.

  6. Reset PCIe Bus:
    Try sudo setpci -s 06:00.0 COMMAND=0x06 to reset and re-enumerate the device.

If these steps don’t resolve the issue, it might be a limitation of the Geekworm PCIe switch. Please let me know the results of your troubleshooting, and we can explore further options if needed.

Best regards,
Omri