Enabling Virtual Machine to access Hailo-8 devices

Guides
1

This guide provides detailed instructions for setting up a Virtual Machine to access Hailo-8 devices. We begin with an overview of the current containerization topologies supported by Hailo.

This guide currently focuses exclusively on QEMU/KVM and does not cover other virtualization methods.

Non VM based topologies

Host Linux OS

This is the simplest and most straightforward topology. The host system runs a Linux distribution as its operating system. The Hailo PCIe driver, Hailo-RT library, and service are installed on the host. With the Hailo-RT service, multiple processes can perform inference on one or more Hailo devices connected to the system. This use-case is illustrated in the figure below:

Host Linux OS
Host Linux OS1058×813 105 KB

Host Linux OS + Docker Container

This topology is similar to the previous one, but with the added flexibility of using Docker containers to package application processes. These containers can include different Linux distributions, compilers, and tool versions than those on the host Linux system. For example, you can create a CentOS or Gentoo Linux container and run it within the host’s Ubuntu OS. The only constraint is that the Linux distribution in the Docker container must share the same Linux kernel as the host. This scenario is illustrated in the figure below:

Host Linux OS + Docker container
Host Linux OS + Docker container1058×813 232 KB

Regarding inferencing on Hailo, this scenario has the following limitations:

  • At any given time, only one Docker container can send inference jobs to the Hailo devices. If you want multiple Docker containers to access Hailo devices, you will need to implement a synchronization method between them. Or one can try to move the Hailo-RT service outside of the docker containers.

  • The docker container must be run with the following options:

    -v /dev:/dev \
-v /lib/firmware:/lib/firmware \
-v /lib/udev/rules.d:/lib/udev/rules.d \
-v /lib/modules:/lib/modules \
--device=/dev/hailo0:/dev/hailo0 \

    When using several Hailo devices, define the additional devices accordingly. For example, for a 2nd device, add:
    --device=/dev/hailo1:/dev/hailo1 \

  • The Hailo PCIe driver must be installed on the host Linux system since Docker containers cannot virtualize peripherals. Consequently, the version of the Hailo-RT library in the container must match the version of the PCIe driver on the host. If this constraint cannot be met, it is recommended to use a Virtual Machine. See the next paragraph for details.

VM based topology

Host Linux OS + Barebone hypervisor + Virtual Machine

Running applications that perform inferencing on Hailo-8 inside a Virtual Machine offers several advantages: ease of use, security, flexibility, and platform independence. This setup allows you to install a Linux distribution with a different kernel in the VM than the host Linux OS. For Hailo, the primary benefit is that the Hailo-RT PCIe driver (v4.20 or above) can be installed inside the Virtual Machine. This makes it possible to create different VMs with various versions of the Hailo-RT PCIe driver, all running on the same Linux host system. This scenario is illustrated in the figure below:

Host Linux OS + Virtual Machine
Host Linux OS + Virtual Machine1115×922 225 KB

The host Linux system does not require the Hailo-8 PCIe driver to be installed. Communication between the VM’s Hailo PCIe driver and the Hailo-8 is facilitated by VFIO-PCI pass-through, an IOMMU/device-agnostic Linux framework that provides secure, IOMMU-protected direct device access to user-space. Note that a PCIe device can only be passed through to one single VM.

Within the Linux virtual machine, Docker containers can be used, although they are not shown in the figure above. The host Linux can also have Hailo-RT and PCIe drivers installed.

Setup

The steps below provide details on how to setup a system using the Linux KVM hypervisor which supports VFIO.

  1. Install QEMU+KVM: How to Create Virtual Machines in Ubuntu Using QEMU/KVM Tool

  2. Enable PCIe pass-through:

    • Enable IOMMU feature
      Enable the IOMMU feature via your grub config.
      sudo nano /etc/default/grub
      Edit the line which starts with GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX to match:
      GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
      GRUB_CMDLINE_LINUX="intel_iommu=on"
      In case you are using an AMD CPU the line should read:GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt"
      GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"
      Once you’re done editing, save the changes and exit the editor (CTRL+x CTRL+y).
      Afterwards run:sudo update-grub
      Reboot the system when the command has finished.
      Verify if IOMMU is enabled by running after a reboot:
      dmesg |grep IOMMU
      [ 0.044930] DMAR: IOMMU enabled
      [ 0.178115] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1

    • Identification of the guest Hailo-8
      The goal is to identify and isolate the Hailo-8 devices before we pass them over to the virtual machine.
      You can easily identify Hailo-8 devices by running:
      lspci -D | grep "Co-processor"
      0000:02:00.0 Co-processor: Device 1e60:2864 (rev 01)
      In the output above, there is a single Hailo-8 device in the system and its ID is 0000:02:00.0 .
      We want to apply the vfio-pci driver to the passed-through Hailo-8s, before the regular Hailo-8 PCIe driver can take control of them. This is the most crucial step in the process.
      Optionally, you can identify the IOMMU grouping of the Hailo-8 devices by running this script:

      #!/bin/bash
# change the 999 if needed
shopt -s nullglob
for d in /sys/kernel/iommu_groups/{0..999}/devices/; do
    n=${d#/iommu_groups/}; n=${n%%/}
    printf 'IOMMU Group %s ' "$n"
    lspci -Dnns "${d##*/}"                                                                         
done

      Here is an example output:

      IOMMU Group 0 0000:00:00.0 Host bridge [0600]: Intel Corporation Device [8086:452e] (rev 01)
IOMMU Group 1 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4571] (rev 01)
IOMMU Group 2 0000:00:08.0 System peripheral [0880]: Intel Corporation Device [8086:4511] (rev 01)
IOMMU Group 3 0000:00:14.0 USB controller [0c03]: Intel Corporation Device [8086:4b7d] (rev 11)
...
IOMMU Group 15 0000:00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:4b00] (rev 11)
IOMMU Group 15 0000:00:1f.3 Audio device [0403]: Intel Corporation Device [8086:4b58] (rev 11)
IOMMU Group 15 0000:00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:4b23] (rev 11)
IOMMU Group 15 0000:00:1f.5 Serial bus controller [0c80]: Intel Corporation Device [8086:4b24] (rev 11)
IOMMU Group 16 0000:02:00.0 Co-processor [0b40]: Device [1e60:2864] (rev 01)
IOMMU Group 17 0000:03:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:0d9f] (rev 03)

      Note that the Hailo device corresponds to Co-processor and belongs to IOMMU Group 16. Knowing that the Hailo-8 devices belong to a IOMMU group confirms that PCIe pass-through can be enabled. In case there are multiple devices in the same IOMMU group, all should be passed through to the same VM. This applies to the Century card use-case, which means that all the devices on a single century card have to be passed to the same VM. One option to work around this limitation is placing the Hailo8 in a different PCIe/M.2 slot in which it will be the only device in the group.

    • Implement the PCIe pass-through
      This can be done by creating the file /etc/initramfs-tools/scripts/init-top/vfio.sh and add the following lines:

      #!/bin/sh
PREREQ=""
prereqs()
{
   echo "$PREREQ"
}
case $1 in
prereqs)
   prereqs
   exit 0
   ;;
esac
for dev in 0000:02:00.0 0000:02:00.0
do
 echo "vfio-pci" > /sys/bus/pci/devices/$dev/driver_override
 echo "$dev" > /sys/bus/pci/drivers/vfio-pci/bind
done
exit 0

      Please make sure to specify the first and last device IDs in the line for dev in 0000:02:00.0 0000:02:00.0 . In the example, we have one Hailo-8 in the system, that’s why the first and last device IDs are the same.
      Make the script executable via:
      sudo chmod +x /etc/initramfs-tools/scripts/init-top/vfio.sh
      Create another file via /etc/initramfs-tools/modules
      And add the following lines:
      options kvm ignore_msrs=1

      Save and close the file.

      When all is done run:

      sudo update-initramfs -u -k all

      Reboot the system.
      Attention though, in case PCI-hardware is added or removed from the system the PCI bus ids will change (also sometimes after BIOS updates). When this happens, run the above step again.

    • Verify the isolation
      In order to verify a proper isolation of the device, run:
      lspci -Dnnv
      find the line "Kernel driver in use" for the Hailo Co-processor, it should state vfio-pci.

      0000:02:00.0 Co-processor [0b40]: Device [1e60:2864] (rev 01)
	Subsystem: Device [1e60:2864]
	Flags: fast devsel, IRQ 255
	Memory at 6000004000 (64-bit, prefetchable) [disabled] [size=16K]
	Memory at 6000008000 (64-bit, prefetchable) [disabled] [size=4K]
	Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci

    • Setting Hailo-8 reset method to null.
      There is one extra step specific to the Hailo device that needs to be done: to prevent the system from freezing when a guest VM is started, the reset method of the device must be changed from “flr bus” to null. This action must be done at every boot of the host machine. This can be done by creating/editing the /etc/rc.local script and adding the following lines to it:

      #!/bin/bash
lspci -D | grep -E "Co-processor: Device 1e60:2864 \(rev 01\)|Hailo" | awk '{print $1}' | while read -r device; do
    echo | sudo tee /sys/bus/pci/devices/$device/reset_method
done

    Make sure you do so as root. Verify that the file is owned by root and has execution permission. Note that script iterates through all the Hailo-8s attached to the system in case there are more than one.

    • Add Hailo-8 as a PCIe device to the VM guest

      • Create and open a Virtual Machine

        image
        image552×588 18.4 KB

      • Click on the Information icon

        image
        image1406×807 104 KB

      • Click on the button ‘Add Hardware’ located on the far bottom left corner of the window. Select PCI Host Device and then lookup in the list the device ID corresponding to Hailo, obtained with lspci. In our example, it is 0000:02:00:0 . Click on Finish.

        image
        image729×604 74.9 KB

      • Run the VM

  • VM Setup
    The VM must have Hailo-RT 4.20 or greater installed.
    If not, you may encounter issues such as the following error message whenhailortcli fw-control identify is executed after booting the VM:
    [HailoRT] [error] CHECK failed - Failed to open device file /dev/hailo0 with error 6
    [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
    [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
    [HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
    [HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
    Please upgrade to Hailo-RT v4.20 or greater if you encounter this issue.
2

Thank you!

Some minor comments:

The script didn’t work for me (the for has a ‘done’ missing amongst other things).
This version may be more useful:

#!/bin/bash
cd /sys/kernel/iommu_groups/
for g in `ls | sort -n`; do
    cd /sys/kernel/iommu_groups/$g/devices/
    for d in *; do
        echo "IOMMU Group $g `lspci -Dnn -s $d`"
    done
done

Also my (working) AMD Ryzen 9 under linux 6.1 gives:

elm:~# dmesg |grep IOMMU
[    0.342229] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.343021] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.405964] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.536789] AMD-Vi: AMD IOMMUv2 loaded and initialized

So I think the ‘AMD-Vi: Found IOMMU cap’ is probably the key message

If using libvirt directly then virsh edit and add an entry under devices:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

for a device showing 0000:0b:00.0