Hi everyone! I’m new to the Hailo community. I’m a French electronics and robotics enthusiast, working on several Linux, embedded systems and AI projects. I recently integrated a Hailo-8 into my 24/7 Proxmox server, and this led me down a rather unexpected debugging journey!
I’m reaching out after experiencing several crashes, PCIe-related issues and successive errors on my Proxmox server running 24/7, which uses a Hailo-8 M.2 PCIe accelerator.
Since this server also hosts other services and virtual machines, reliability is particularly important to me. Having to deal with unexpected crashes and system instability led me to investigate the possible causes rather than simply restarting everything and hoping for the best.
My environment
- **Host:** Proxmox VE 9
- **Guest:** Debian 13 (UEFI, Secure Boot enabled)
- **Accelerator:** Hailo-8 M.2 PCIe (VFIO passthrough)
- **HailoRT:** 4.24.0 (built from source)
- **Kernel driver:** Hailo PCIe 4.24.0 (built from source using DKMS, locally signed with an enrolled MOK key)
- **Firmware:** Hailo-8 firmware 4.24.0 (official binary)
- **GPU:** NVIDIA RTX A2000 (PCIe passthrough)
What I have investigated so far
My initial investigations focused on PCIe passthrough, device initialization and reset behavior.
One particularly serious issue involved a complete Proxmox host freeze during Hailo-8 initialization. After investigating the PCIe reset mechanisms, switching the device’s reset method to bus allowed the VM to start successfully.
This improved the situation, but it also encouraged me to examine the HailoRT userspace library and Linux PCIe driver more deeply.
During this work, I identified several potential problems involving:
-
Shutdown error propagation and preservation of failure status.
-
DMA mapping and unmapping error handling.
-
Buffer ownership and resource lifetime.
-
Concurrent operations during device removal.
-
Synchronization between the host driver and the device firmware.
I have developed several isolated experimental corrections and differential tests to better understand these behaviors.
Current results
After stabilizing the passthrough configuration, I performed hardware validation on the actual Hailo-8.
So far:
-
Nine real inference benchmarks completed successfully.
-
Three separate HailoRT variants tested.
-
Stable inference throughput of approximately 1,037 FPS with YOLOv8n.
-
No new kernel messages collected during these benchmarks.
-
Experimental shutdown-related fixes passed targeted differential tests.
-
A small kernel correction addressing two ioctl/mmap admission checks passed 21/21 simulated test cases, compared with 19/21 for the original implementation.
-
Multiple kernel module builds completed successfully.
Importantly, these results do not prove that all the initial crashes have been resolved, nor that every source-code issue identified was responsible for them.
Some findings concern error paths and resource-lifetime guarantees that remain difficult to validate without additional information about the firmware.
Where I need help from Hailo
My goal is to make the Hailo-8 reliable for continuous operation, while contributing useful corrections upstream whenever possible.
I would particularly appreciate clarification on the following questions:
1. PCIe reset and virtualization
What reset method does Hailo officially recommend when using a Hailo-8 with VFIO PCIe passthrough?
Are there known limitations involving FLR, bus reset or device reinitialization?
2. DMA and resource ownership
What guarantees that the firmware has stopped accessing DMA buffers or descriptors before the host driver releases or reuses those resources?
Does a successful shutdown, cancellation or reset acknowledgment guarantee that no further DMA activity can occur?
3. DMA unmapping failures
What is the expected recovery behavior if an unmap operation fails? Which layer is responsible for retaining ownership of the affected resources?
4. Device removal and concurrent operations
What synchronization guarantees are expected between device removal, ioctl/mmap operations and the final destruction of driver resources?
5. Driver and firmware documentation
Is there technical documentation describing these internal contracts, or a recommended way to discuss potential kernel driver fixes with Hailo engineers?
My intention
I am not trying to claim that every identified issue is necessarily a bug affecting normal operation.
I would like to distinguish confirmed problems from potential weaknesses, validate proposed changes properly, and avoid introducing unsafe modifications to DMA or resource management.
I have preserved detailed investigation reports, reproducible tests, source comparisons and experimental patches that I would be happy to share.
My objective is simple: improve the reliability of the Hailo-8 in long-running systems and, hopefully, contribute those improvements back to the project.
Any guidance or feedback from the Hailo engineering team would be greatly appreciated.
Thank you!