Nvidia-dali-tf-plugin

General
1

During optimization level 3 using Adaround, I’m getting the following warning:

Warning: DALI is not installed, using TensorFlow dataset for layer-by-layer training. Using DALI will significantly improve training time. To install it, use:
pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-cuda110 nvidia-dali-tf-plugin-cuda110

I attempted to install DALI, but it failed because I first tried using the link in the warning, but it had changed. Then, I found this one on the Nvidia page:

pip install nvidia-dali-tf-plugin-cuda110

When I try to install it, I’m getting the following error:

Build from source failed with error: Command ‘[‘g++’, ‘-Wl,-R,$ORIGIN/…’, ‘-std=c++14’, ‘-DNDEBUG’, ‘-shared’, ‘/tmp/dali_stub_2pz0lpb6/dali_stub.cc’, ‘-o’, ‘/tmp/dali_stub_2pz0lpb6/libdali.so’, ‘-fPIC’, ‘-I/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/include’, ‘-I/home/admin/anaconda3/envs/conda_env/nvvm/include’, ‘-O2’]’ returned non-zero exit status 1.
Traceback (most recent call last):
File “”, line 2, in
File “”, line 34, in
File “/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/setup.py”, line 37, in
setup(name=‘nvidia-dali-tf-plugin-cuda110’,
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/init.py”, line 107, in setup
return distutils.core.setup(**attrs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/core.py”, line 185, in setup
return run_commands(dist)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/core.py”, line 201, in run_commands
dist.run_commands()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/dist.py”, line 969, in run_commands
self.run_command(cmd)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/dist.py”, line 1234, in run_command
super().run_command(command)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/dist.py”, line 988, in run_command
cmd_obj.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/wheel/_bdist_wheel.py”, line 387, in run
self.run_command(“build”)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py”, line 318, in run_command
self.distribution.run_command(command)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/dist.py”, line 1234, in run_command
super().run_command(command)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/dist.py”, line 988, in run_command
cmd_obj.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/command/build.py”, line 131, in run
self.run_command(cmd_name)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py”, line 318, in run_command
self.distribution.run_command(command)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/dist.py”, line 1234, in run_command
super().run_command(command)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/setuptools/_distutils/dist.py”, line 988, in run_command
cmd_obj.run()
File “/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/setup.py”, line 25, in run
helper.install()
File “/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/dali_tf_plugin_install_tool.py”, line 436, in install
raise e
File “/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/dali_tf_plugin_install_tool.py”, line 428, in install
self.build()
File “/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/dali_tf_plugin_install_tool.py”, line 356, in build
subprocess.check_call(cmd, cwd=self.src_path, shell=False) # nosec B603
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/subprocess.py”, line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘g++’, ‘-Wl,-R,$ORIGIN/…’, ‘-std=c++14’, ‘-DNDEBUG’, ‘-shared’, ‘/tmp/dali_stub_2pz0lpb6/dali_stub.cc’, ‘-o’, ‘/tmp/dali_stub_2pz0lpb6/libdali.so’, ‘-fPIC’, ‘-I/tmp/pip-install-5ntjmzsa/nvidia-dali-tf-plugin-cuda110_845d47d3b4434259a213986eeecfeba2/include’, ‘-I/home/admin/anaconda3/envs/conda_env/nvvm/include’, ‘-O2’]’ returned non-zero exit status 1.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for nvidia-dali-tf-plugin-cuda110
Running setup.py clean for nvidia-dali-tf-plugin-cuda110
Failed to build nvidia-dali-tf-plugin-cuda110
ERROR: Failed to build installable wheels for some pyproject.toml based projects (nvidia-dali-tf-plugin-cuda110)

Has anyone encountered this issue or have any suggestions on how to resolve it?

2

I was able to resolve the problem using the following command:

CPATH=/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/tensorflow/include/third_party/gpus/cuda/include pip install --extra-index-url https://pypi.nvidia.com nvidia-dali-tf-plugin-cuda110

However, this installs the latest version of nvidia-dali-tf, which is not compatible with TensorFlow 2.12.

The URL https://developer.download.nvidia.com/compute/redist should contain legacy versions, but the files are no longer there.

3

I managed to install nvidia-dali-tf-plugin-cuda110 using a previous version:

# For TensorFlow 2.12, you need DALI 1.25+ 
# Install DALI without TF plugin first 
pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-cuda110==1.25.0 

# Then try the TF plugin for newer version 
pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-tf-plugin-cuda110==1.25.0

This works and there are no WARNINGS in the logs, but there is no significant performance improvement from using it. The conversion time per layer has decreased from 20 minutes to 16 minutes for the same YOLO model.

1 Like