Something I would like to share with the Hailo community after many attempts and many hours… 
Just take note that you will need “hailo_dataflow_compiler-3.33.0-py3-none-linux_x86_64.whl“ available as a download. See in the script below.
On Vast.AI create a template with the following options:
Vast.AI Template:
Docker image path/tag: nvidia/cuda:12.5.1-devel-ubuntu22.04
Docker options: -p 22:22 -p 8888:8888 -p 6006:6006 --gpus all --shm-size=64g
After the server has started from the Vast.AI console, run the following bash script to install the correct environment to run Hailo Dataflow Compiler 3.33, as well as Ultralytics’s Yolo. It will install the correct versions for CUDA, CUDNN and DALI. Depending on the server hardware and bandwidht, it takes about 3-5 minutes.
#!/bin/bash
set -euo pipefail
echo “=== Hailo DFC Provisioning Starting ===”
apt-get update && apt-get upgrade -y
apt-get install -y
build-essential
python3-dev
python3-venv
python3-tk
python3-pip
graphviz
libgraphviz-dev
unzip
wget
curl
git
vim
htop
libgl1-mesa-glx
libglib2.0-0
libfuse2
mkdir -p /workspace
cd /workspace
python3 -m venv hailo_env --system-site-packages
. hailo_env/bin/activate
pip install --upgrade pip wheel setuptools
pip install --extra-index-url https://pypi.nvidia.com nvidia-dali-cuda120
sudo apt install nano -y
wget https://developer.download.nvidia.com/compute/cudnn/9.10.0/local_installers/cudnn-local-repo-ubuntu2204-9.10.0_1.0-1_amd64.deb
dpkg -i cudnn-local-repo-ubuntu2204-9.10.0_1.0-1_amd64.deb
sudo cp /var/cudnn-local-repo-ubuntu2204-9.10.0/cudnn--keyring.gpg /usr/share/keyrings/
dpkg -i cudnn-local-repo-ubuntu2204-9.10.0_1.0-1_amd64.deb
apt-get update
apt-get install -y libcudnn9-cuda-12 libcudnn9-dev-cuda-12 libcudnn9-static-cuda-12
wget https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/linux-x86_64/cudnn-linux-x86_64-9.10.0.56_cuda12-archive.tar.xz
tar -xvf cudnn-linux-x86_64-9.10.0.56_cuda12-archive.tar.xz
cp -r cudnn-linux-x86_64-9.10.0.56_cuda12-archive/include/ /usr/local/cuda/include/
cp -r cudnn-linux-x86_64-9.10.0.56_cuda12-archive/lib/* /usr/local/cuda/lib64/
chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*
ldconfig
pip install nvidia-dali-cuda120==1.51.*
wget --no-check-certificate --user=admin --password=Admin001 https:///hailo_dataflow_compiler-3.33.0-py3-none-linux_x86_64.whl
pip install hailo_dataflow_compiler-3.33.0-py3-none-linux_x86_64.whl
pip install nvidia-dali-tf-plugin-cuda120==1.51.* --no-build-isolation
#Your downloads can follow here, eg. wget --no-check-certificate --user=admin --password=Admin_001 https://10.199.103.14:4443/best.har
hailo
echo “=== Hailo provisioning COMPLETE ===”
echo “Workspace ready at /workspace”
echo “Activate with: source /workspace/hailo_env/bin/activate”
echo “Next: scp your .whl file here and pip install it.”