Setup:
- Host: Proxmox VE 9.2.4 (kernel 7.0.14-4-pve), AMD Ryzen 9 7900, IOMMU (AMD-Vi) active
- Hailo-8 M.2 module, in its own clean IOMMU group, passed through to a Debian 13 (trixie) guest VM via VFIO
- Guest kernel: 6.12.95+deb13-cloud-amd64
- hailort-drivers built from source at tag v4.19.0 (matched to HailoRT userspace library v4.19.0, confirmed via
ghcr.io/blakeblackshear/frigate:stable-h8l)
Symptom:
The driver probes and loads firmware successfully on every boot without exception:
hailo 0000:01:00.0: Firmware loaded successfully
hailo 0000:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0
However, any subsequent attempt to open /dev/hailo0 — whether via hailortcli fw-control identify inside the Frigate stable-h8l container, or the same binary/library extracted and run natively on the guest with zero Docker involvement — fails 100% of the time with:
[HailoRT] [error] CHECK failed - Failed to open device file /dev/hailo0 with error 6
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
Guest dmesg at the moment of failure:
hailo 0000:01:00.0: Device disconnected while opening device
hailo 0000:01:00.0: Refused to change power state from D0 to D3hot
Traced this to hailo_pcie_is_device_connected() in common/pcie_common.c:
bool hailo_pcie_is_device_connected(struct hailo_pcie_resources *resources)
{
return PCI_VENDOR_ID_HAILO == hailo_resource_read16(&resources->config, PCIE_CONFIG_VENDOR_OFFSET);
}
This is returning false — i.e., a PCI config space read of the Vendor ID at that moment isn’t returning 0x1e60 — despite the device being fully functional moments earlier during probe.
What’s been ruled out:
- Host-side: no PCIe link-down, hotplug, or reset event visible in host dmesg at the moment of guest-side failure (checked live with
dmesg -w during the test)
- ASPM: already fully disabled at the hardware level (
LnkCtl: ASPM Disabled per lspci -vv)
options vfio-pci ids=1e60:2864 disable_vfio_pci_flr=1 — no change- Adding
disable_idle_d3=1 to the same options — no change
- Switching from Proxmox’s simple
hostpci0 mapping to a manually defined dedicated PCIe root port with hotplug=off — no change, identical failure either way
- Docker vs. native execution — identical failure both ways, ruling out container/cgroup device access as a factor
The failure is fully deterministic (not intermittent) across every configuration tried. Would appreciate any insight into what else might cause PCIE_CONFIG_VENDOR_OFFSET to read incorrectly specifically on open (not probe) under VFIO passthrough, or any additional debug flags/logging we could enable on the driver side to narrow this further.
Update: resolved. Leaving the full writeup here since a few people have viewed this without a fix landing.
Short version: upgrade past v4.19.0. We landed on v4.21.0 and separately confirmed v4.24.0 — both work cleanly on the exact same VFIO passthrough setup that failed deterministically on v4.19.0. Root cause was never bisected to a specific commit, but the changelog shows v4.20.0 was the first release to add “PCIe passthrough for Hailo-8 (preview)” support, which lines up exactly with our symptom (device reachable at boot-time probe, fails specifically on open() after that — i.e., a reconnection/re-enumeration path that passthrough-aware code needs to handle differently than bare-metal).
Environment (unchanged from original post): Proxmox VE, VFIO PCIe passthrough of a Hailo-8 M.2 module into a Debian 13 guest, hostpci0 mapping, no custom PCIe topology needed in the end.
What actually fixed it:
-
Stopped building the driver from source (git clone hailort-drivers, make, make install). That’s not the documented install path and we can’t fully rule out it contributing to the original failure, though the version-boundary evidence points more strongly at v4.19.0 itself.
-
Installed via the official combined .deb path instead, which registers the driver with DKMS:
sudo dpkg --install hailort_<version>_amd64.deb hailort-pcie-driver_<version>_all.deb
Reboot required afterward. 3. Gotcha along the way: if dkms isn’t already installed on your system, this silently falls back to a non-DKMS one-off build with no clear error — just a line easy to miss: Failed to install PCIe driver to the DKMS tree. Trying to install PCIe driver without DKMS. Install dkms first (apt-get install dkms), then reinstall the .deb so it registers properly. Confirm with dkms status — should show hailo_pci/<version>, <kernel>, x86_64: installed. 4. Verified with both hailortcli scan and hailortcli fw-control identify — both came back clean at v4.21.0 and v4.24.0, versus the deterministic HAILO_DRIVER_FAIL / “Device disconnected while opening device” we got on every configuration at v4.19.0 (native, Docker, simple hostpci0 mapping, dedicated PCIe root port with hotplug=off, ASPM disabled, disable_vfio_pci_flr/disable_idle_d3 VFIO options — none of it mattered at v4.19.0).
If you’re running this behind Frigate specifically: the official stable-h8l image tag may still bundle an old hailort Python library (we found 4.19.0 baked in) even after you fix the host driver — you’ll just get the version-mismatch error inverted. Frigate’s own hailo build pin lives in docker/main/install_hailort.sh in their repo (or docker/hailo8l/install_hailort.sh on older tags) — match your host driver version there and build locally with make local if upstream hasn’t caught up yet.