Multiprocess inference in Docker

General
1

There is no documentation how to run inference inside a Docker container.
I have understood that hailort should be installed on the host, and the host should run hailort.service so it can schedule models.

The docker container has installed pyhailort wheel.

When running the container it is unable to find libhailort

ImportError: libhailort.so.4.20.0: cannot open shared object file: No such file or directory

Following this thread I have mapped the volumes into the container but it produces the same result: Issue with Hailo GStreamer Pipeline on reComputer AI R2000: Multi-process-service Error

I have taken libhailort.so libhailort.so.4.20.0 and hailortcli and copied them INTO the container at /usr/local/lib and /usr/local/bin
pyhailo is then able to find libhailort but fails because HailoRT service is not active

[HailoRT] [error] CHECK_GRPC_STATUS failed with error code: 14.
[HailoRT] [warning] Make sure HailoRT service is enabled and active!

The HailoRT service is in fact running on the host.

This has been very frustrating, can you please share a working Dockerfile

2

Hey @axel.moller ,

For a pre-built docker image , please check out our AI Docker Suite which includes tappas. ( 4.20 version 01-2025 ) SW-downloads

Now for your issue :

The HailoRT service communicates via a Unix domain socket located at:

/var/run/hailo/hailort-service.sock

If this socket path isn’t mounted into the container, pyhailort can’t talk to the service — resulting in:

[HailoRT] [error] CHECK_GRPC_STATUS failed with error code: 14.

Even if you’ve copied libhailort.so and hailortcli, that doesn’t establish communication with the host-side service.

To test this run :

hailortcli fw-control identify

Check that /var/run/hailo/hailort-service.sock exists on the host:

ls -l /var/run/hailo/

and maybe add this to the docker

  -v /var/run/hailo:/var/run/hailo \
3

Thank you @omria

There is no hailort-service.sock file in /var/run/hailo
I did however find a hailort_uds.sock file in /tmp. Mapping it as a volume in the container solved the communication issue with the host.

$ ls /var/run/hailo/
hailort_service.pid
$ ls /tmp/
hailort-service
hailort_uds.sock
[...]
$ ls /tmp/hailort-service/

I have installed Hailo via dpkg --install hailort-pcie-driver_4.20.0_all.deb on the host.
What could explain the discrepancies of where the .sock file is located?

4

Hey @axel.moller,

Yeah, this changed between versions 4.17 and 4.18. The default location for the HailoRT and driver UDS (Unix Domain Socket) moved in different releases:

  • Older versions (up to around v4.17) used: /var/run/hailo/hailort-service.sock
  • Newer versions (v4.18 and up) usually use: /tmp/hailort_uds.sock or /tmp/hailort-service/hailort_uds.sock

Hope that clears it up!