Hailo-10H detected by lspci but not by HailoRT 5.4.0

General Discussion
1

Hello,

We have one Hailo-10H module that is visible in lspci as a PCIe device, but it is not recognized by HailoRT 5.4.0 and does not appear in hailortcli scan.

Other Hailo-10H modules work normally on the same host and PCIe environment.

Could you please confirm:

  • whether there is any known issue or reported case where a Hailo-10H device is detected by lspci but fails to initialize with HailoRT 5.4.0;

  • whether there is any recommended fix, firmware update, or recovery procedure for this case.

If needed, we can provide the lspci, dmesg, and Hailo driver logs for the affected module.

Thank you.

Best regards,
Jusung Kang

Additional update:

We reproduced the issue with the affected Hailo-10H 4GB module on the same PCIe slot and host where another module works correctly.

The device is visible in lspci as:



01:00.0 Co-processor: Hailo Technologies Ltd. Device 45c4 (rev ff)

However, HailoRT does not successfully initialize it. The Hailo 5.4.0 driver proceeds through PCIe BAR mapping, loads customer_certificate.bin and scu_fw.bin, reads Board SKU-ID: 1, and successfully transfers u-boot-1.dtb.signed.

The initialization then fails with:



Timeout waiting for firmware file
Failed writing SOC firmware on stage 2
Firmware load failed
Failed activating board -110
probe of 0000:01:00.0 failed with error -110

As a result, the device does not appear in hailortcli scan / runtime access, even though PCIe enumeration itself succeeds.

Could you please confirm whether this Stage 2 firmware boot timeout (-110) is a known issue, and whether there is any recommended firmware recovery, update, or diagnostic procedure for this case?

One additional point: this affected module is reported as rev ff by lspci, which differs from the working modules we tested.

Additional Info

Device enumerates normally as PCIe rev 01 after a cold boot.
During Hailo-10H Stage 2 boot, after u-boot-1.dtb.signed is transferred, the device becomes non-responsive. Its PCI configuration space reads as 0xff, the upstream link falls from 8 GT/s to 2.5 GT/s, and it does not recover after 30 seconds. The same behavior was also observed with HailoRT 5.3.

Additional debugging update

We performed further testing with the affected Hailo-10H 4GB module only, using the same host and PCIe slot where other Hailo-10H modules work normally.

After a full power cycle, with the Hailo driver blacklisted, the affected module enumerates normally:



01:00.0 Co-processor [0b40]: Hailo Technologies Ltd. Device [1e60:45c4] (rev 01)

The PCIe link also comes up normally at 8.0 GT/s x2, with no obvious PCIe/AER errors at this stage.

We then manually load the HailoRT 5.4.0 PCIe driver. The initialization proceeds successfully through:



customer_certificate.bin              OK
scu_fw.bin                            OK
Board SKU-ID: 1                       OK
u-boot-1.dtb.signed                   transferred successfully

However, immediately afterwards, Stage 2 firmware boot fails:



Timeout waiting for firmware file
Failed writing SOC firmware on stage 2
SCU log could not be read from device
Firmware load failed
Failed activating board -110
probe of 0000:01:00.0 failed with error -110

After this failure, the PCIe endpoint itself becomes non-responsive. The PCI configuration space starts returning 0xff:



01:00.0 ... (rev ff)
Revision ID: ff

The upstream Root Port also changes from 8.0 GT/s x2 to:



LnkSta: Speed 2.5GT/s (downgraded), Width x2
DevSta: CorrErr+

The device does not recover after waiting, and hailortcli scan / monitor report no Hailo device. No /dev/h1x* device node is created.

We also tested PCIe recovery:



reset_method: flr bus

but both the endpoint reset and the upstream reset_subordinate operation fail with:



Inappropriate ioctl for device

The device remains at rev ff until a full power cycle, after which it returns to rev 01.

We have also observed the same Stage 2 timeout behavior with HailoRT 5.3.0, so this does not appear to be specific to HailoRT 5.4.0.

Also, this module is installed on the 4GB slot and follows the normal SKU-ID 1 → u-boot-1.dtb.signed path; no SKU/DTB override is applied to this device.

Based on these results, the issue appears to occur during the device-side boot transition after the SKU-1 DTB is transferred, rather than during the initial PCIe enumeration.

Could you please advise:

  • whether this Stage 2 failure followed by loss of PCIe configuration-space access is a known Hailo-10H issue;

  • whether it may indicate a DDR initialization, bootloader, PMIC/power, or device provisioning problem;

  • whether there is any recovery/reflash procedure we can try;

  • or whether this module should be considered for RMA/replacement?

We can provide the full dmesg, lspci -vvv, and PCIe link logs if needed.