Hailo‑10H on RK3588 (OrangePi 5 Max): device wedges with HAILO_VDMA_LAUNCH_TRANSFER failed with 5, PCIe link drops 8.0 → 2.5 GT/s, only a PCI remove/rescan recovers it.
On an OrangePi 5 Max (RK3588), a Hailo‑10H M.2 module wedges within ~20 seconds of a real application starting inference. The kernel logs Failed to launch transfer, userspace gets HAILO_DRIVER_OPERATION_FAILED(36), and the PCIe link retrains down from 8.0 GT/s to 2.5 GT/s. A modprobe -r / modprobe cycle does not recover it — the reload fails at probe with “Failed reading device BARs, device may be disconnected”. Only a PCI remove + rescan brings it back, which also restores the link to 8.0 GT/s until the next run.
System information
Board: RK3588 OPi 5 Max
OS: Orange Pi 1.0.0 Jammy (Ubuntu 22.04)
Kernel: 6.1.43-rockchip-rk3588, aarch64
RAM: 15 GiB CMA: 131072 kB
HailoRT: hailort 5.3.0, hailort-pcie-driver 5.3.0, python wheel hailort 5.3.0 (cp310)
DKMS: hailo1x_pci/5.3.0, 6.1.43-rockchip-rk3588, aarch64: installed
vermagic: 6.1.43-rockchip-rk3588 SMP mod_unload aarch64 (matches uname -r)
Device: HAILO10H, firmware 5.3.0 (release,app), node /dev/h1x-0
PCI ID: 0000:01:00.0 Co-processor [0b40]: Hailo Technologies [1e60:45c4] (rev 01)
Controller: platform/fe150000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0
(rockchip,rk3588-pcie / snps,dw-pcie — the pcie3x4 controller)
Root port: 0000:00:00.0 Rockchip [1d87:3588] (rev 01)
PCIe state (while wedged)
Endpoint 0000:01:00.0
DevCap: MaxPayload 256 bytes
DevCtl: MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+
LnkCap: Speed 8GT/s, Width x4, ASPM L0s L1
LnkCtl: ASPM Disabled
LnkSta: Speed 2.5GT/s (downgraded), Width x4 (ok)
DevCap2: Completion Timeout: Not Supported, TimeoutDis+
CESta: AdvNonFatalErr+
Root port 0000:00:00.0
DevCtl: MaxPayload 128 bytes, MaxReadReq 512 bytes
LnkCap: Speed 8GT/s, Width x4, ASPM L1
LnkCtl: ASPM Disabled
LnkSta: Speed 2.5GT/s (downgraded), Width x4 (ok)
UESta: CmpltTO+ <-- Completion Timeout
CESta: RxErr+
ASPM is already disabled on both ends (the driver disables L0s at probe). MPS is matched at 128 bytes on both ends. Completion Timeout: Not Supported means the CTO range is not programmable on this hardware.
dmesg
Probe and firmware load always succeed:
hailo1x 0000:01:00.0: SOC Firmware Batch loaded successfully
hailo1x 0000:01:00.0: Firmware loaded in 949 ms
hailo1x 0000:01:00.0: Device created at /dev/h1x-0
Then, shortly after the application starts:
hailo1x 0000:01:00.0: Failed to launch transfer (x many)
hailo1x 0000:01:00.0: Device disconnected while opening device
Userspace
[HailoRT] [error] Ioctl HAILO_VDMA_LAUNCH_TRANSFER failed with 5. Read dmesg log for more info
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed launch transfer
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) and later, once the device is gone:
[HailoRT] [error] CHECK failed - Timeout waiting for transfer completion
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK failed - Failed to open device file /dev/h1x-0 with error 6
Application that triggers it
Python (HailoRT 5.3.0 wheel), two models on a shared VDevice:
params = VDevice.create_params()
params.scheduling_algorithm = HailoSchedulingAlgorithm.ROUND_ROBIN
params.group_id = "SHARED"
vdevice = VDevice(params) # one per model (yolo26m + yolov8m_pose)
...
configured_model.run_async(bindings_list, callback)