Hailo-8 PCIe – Repeated Proxmox host crashes, driver investigation and proposed fixes

Hi everyone! I’m new to the Hailo community. I’m a French electronics and robotics enthusiast, working on several Linux, embedded systems and AI projects. I recently integrated a Hailo-8 into my 24/7 Proxmox server, and this led me down a rather unexpected debugging journey!

I’m reaching out after experiencing several crashes, PCIe-related issues and successive errors on my Proxmox server running 24/7, which uses a Hailo-8 M.2 PCIe accelerator.

Since this server also hosts other services and virtual machines, reliability is particularly important to me. Having to deal with unexpected crashes and system instability led me to investigate the possible causes rather than simply restarting everything and hoping for the best.

My environment

- **Host:** Proxmox VE 9

  • **Guest:** Debian 13 (UEFI, Secure Boot enabled)
  • **Accelerator:** Hailo-8 M.2 PCIe (VFIO passthrough)
  • **HailoRT:** 4.24.0 (built from source)
  • **Kernel driver:** Hailo PCIe 4.24.0 (built from source using DKMS, locally signed with an enrolled MOK key)
  • **Firmware:** Hailo-8 firmware 4.24.0 (official binary)
  • **GPU:** NVIDIA RTX A2000 (PCIe passthrough)

What I have investigated so far

My initial investigations focused on PCIe passthrough, device initialization and reset behavior.

One particularly serious issue involved a complete Proxmox host freeze during Hailo-8 initialization. After investigating the PCIe reset mechanisms, switching the device’s reset method to bus allowed the VM to start successfully.

This improved the situation, but it also encouraged me to examine the HailoRT userspace library and Linux PCIe driver more deeply.

During this work, I identified several potential problems involving:

  • Shutdown error propagation and preservation of failure status.

  • DMA mapping and unmapping error handling.

  • Buffer ownership and resource lifetime.

  • Concurrent operations during device removal.

  • Synchronization between the host driver and the device firmware.

I have developed several isolated experimental corrections and differential tests to better understand these behaviors.

Current results

After stabilizing the passthrough configuration, I performed hardware validation on the actual Hailo-8.

So far:

  • Nine real inference benchmarks completed successfully.

  • Three separate HailoRT variants tested.

  • Stable inference throughput of approximately 1,037 FPS with YOLOv8n.

  • No new kernel messages collected during these benchmarks.

  • Experimental shutdown-related fixes passed targeted differential tests.

  • A small kernel correction addressing two ioctl/mmap admission checks passed 21/21 simulated test cases, compared with 19/21 for the original implementation.

  • Multiple kernel module builds completed successfully.

Importantly, these results do not prove that all the initial crashes have been resolved, nor that every source-code issue identified was responsible for them.

Some findings concern error paths and resource-lifetime guarantees that remain difficult to validate without additional information about the firmware.

Where I need help from Hailo

My goal is to make the Hailo-8 reliable for continuous operation, while contributing useful corrections upstream whenever possible.

I would particularly appreciate clarification on the following questions:

1. PCIe reset and virtualization

What reset method does Hailo officially recommend when using a Hailo-8 with VFIO PCIe passthrough?

Are there known limitations involving FLR, bus reset or device reinitialization?

2. DMA and resource ownership

What guarantees that the firmware has stopped accessing DMA buffers or descriptors before the host driver releases or reuses those resources?

Does a successful shutdown, cancellation or reset acknowledgment guarantee that no further DMA activity can occur?

3. DMA unmapping failures

What is the expected recovery behavior if an unmap operation fails? Which layer is responsible for retaining ownership of the affected resources?

4. Device removal and concurrent operations

What synchronization guarantees are expected between device removal, ioctl/mmap operations and the final destruction of driver resources?

5. Driver and firmware documentation

Is there technical documentation describing these internal contracts, or a recommended way to discuss potential kernel driver fixes with Hailo engineers?

My intention

I am not trying to claim that every identified issue is necessarily a bug affecting normal operation.

I would like to distinguish confirmed problems from potential weaknesses, validate proposed changes properly, and avoid introducing unsafe modifications to DMA or resource management.

I have preserved detailed investigation reports, reproducible tests, source comparisons and experimental patches that I would be happy to share.

My objective is simple: improve the reliability of the Hailo-8 in long-running systems and, hopefully, contribute those improvements back to the project.

Any guidance or feedback from the Hailo engineering team would be greatly appreciated.

Thank you!

Update — November 5, 2026: Several fixes submitted upstream

Hi everyone!

A quick update on my investigation since my original post.

I’ve continued analyzing HailoRT 4.24.0 and the Hailo PCIe driver, and I’ve now submitted six pull requests to the official Hailo repositories, addressing several issues involving shutdown error handling, firmware resets, timeouts and DKMS management.

HailoRT:

Hailo PCIe driver:

These corrections address concrete weaknesses in areas where users have reported difficulties over time, although they are not necessarily the root cause of every reported issue.

All six PRs are still open and awaiting review and approval from Hailo maintainers. None of these fixes should be considered officially integrated yet.

I’ve also completed additional regression tests, real DKMS installation/removal testing and hardware validation.

The latest independently reviewed Hailo-8 integration test successfully processed 31,111 YOLOv8n frames at approximately 1,037 FPS, with Secure Boot enabled and no new matching errors observed in the captured guest kernel logs.

This test used a combined experimental driver with two additional private safeguards. It confirms nominal operation of that composition, but does not independently validate each correction or the firmware reset paths.

Still waiting for technical clarification

I also want to emphasize that I’m still waiting for answers to the technical questions raised in my original post.

In particular, I need clarification regarding:

  • Firmware/PCIe reset behavior and recovery guarantees

  • DMA ownership and confirmation that transfers have actually stopped

  • Safe handling of DMA unmapping failures

  • Synchronization during device removal and concurrent operations

These questions are essential for continuing work on the more critical parts of the driver.

I don’t want to introduce modifications to sensitive DMA, interrupt or reset mechanisms based on assumptions, especially when an incorrect implementation could compromise host stability.

I’m happy to continue investigating, testing and submitting corrections, but further progress on these critical issues requires technical guidance from Hailo engineers.

Any feedback on the open PRs or answers to the outstanding questions would be greatly appreciated.

Thanks again!

Follow-up — Request for technical documentation

Hi Hailo team,

I’m following up on my previous messages, as I haven’t received any technical feedback yet.

I understand that investigating these issues may require time from your engineering team.

However, even without an immediate review of the six submitted pull requests, access to the relevant technical documentation would already help me move forward significantly.

In particular, I’m looking for documentation covering:

  • Hailo-8 PCIe reset sequences and firmware acknowledgment semantics.

  • DMA ownership, cancellation, and completion guarantees.

  • Firmware/host-driver synchronization and resource lifetime requirements.

  • Device removal, concurrent operations, and safe recovery procedures.

I’m not requesting proprietary firmware source code. What I need is a clear description of the hardware/firmware interface contracts required to implement and validate safe kernel driver corrections.

If this documentation is not publicly available, could you please indicate whether it can be shared privately, under an NDA if necessary, or whether an engineering contact could provide the required clarifications?

I’d really like to continue contributing improvements upstream, but I prefer not to make potentially unsafe assumptions about undocumented firmware behavior.

Thank you for your time and consideration!