DFC in Conda env

General
1

I installed the Hailo DFC inside a Conda environment to better manage CUDA versions. My global environment is running the latest version of CUDA, but in my Conda environment, I have CUDA 11.8 and cuDNN 8.6 running, as required by DFC. I tested TensorFlow 2.12, and it is recognizing the GPU correctly (I had this issue before).

However, when I run the optimization using the following code:

import os
from hailo_sdk_client import ClientRunner

# Paths
har_path = "yolov11.har"
calib_dataset_path = "calib_set.npy"
alls_path = "quant.alls"  # Model script: modifies DFC's default behavior.

# Ensure the HAR file exists
assert os.path.isfile(har_path), "Please provide a valid path for the HAR file"

model_name = os.path.splitext(os.path.basename(har_path))[0]

runner = ClientRunner(har=har_path)

# Load the model script into ClientRunner so it is considered during optimization
runner.load_model_script(alls_path)

# Perform the optimization process
runner.optimize(calib_dataset_path)

# Save the optimized model as a Quantized HAR file
quantized_model_har_path = f"{model_name}_quantized_model.har"
runner.save_har(quantized_model_har_path)

I get the following error during the first epoch:

3 root error(s) found.
  (0) INTERNAL:  libdevice not found at ./libdevice.10.bc
         [[{{node update_1_1/StatefulPartitionedCall}}]]
         [[update_2_144/AssignAddVariableOp/_2781]]
         [[group_deps/_2789]]
  (1) INTERNAL:  libdevice not found at ./libdevice.10.bc
         [[{{node update_1_1/StatefulPartitionedCall}}]]
         [[update_2_144/AssignAddVariableOp/_2781]]
  (2) INTERNAL:  libdevice not found at ./libdevice.10.bc
         [[{{node update_1_1/StatefulPartitionedCall}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_582293]

How can I resolve this issue?

Thanks!

3

This is the full error message:

Epoch 1/4
Traceback (most recent call last):
File “/home/admin/dani/4_optimization.py”, line 21, in
runner.optimize(calib_dataset_path)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2128, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1970, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1125, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1319, in _full_acceleras_run
optimization_flow.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 335, in run
step_func()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 111, in parent_wrapper
raise SubprocessTracebackFailure(*child_messages)
hailo_model_optimization.acceleras.utils.acceleras_exceptions.SubprocessTracebackFailure: Subprocess failed with traceback

Traceback (most recent call last):
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 360, in step2
self.post_quantization_optimization()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 422, in post_quantization_optimization
self._finetune()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 712, in _finetune
finetune.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 54, in run
return super().run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 150, in run
self._run_int()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 387, in _run_int
self.run_qft(self._model_native, self._model, metrics=self.metrics)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 507, in run_qft
self.main_train_summary_per_epoch = qft_distiller.fit(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/tensorflow/python/eager/execute.py”, line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node ‘update_1_1/StatefulPartitionedCall’ defined at (most recent call last):
File “/home/admin/dani/4_optimization.py”, line 21, in
runner.optimize(calib_dataset_path)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2128, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1970, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1125, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1319, in _full_acceleras_run
optimization_flow.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 335, in run
step_func()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 94, in parent_wrapper
proc.start()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 121, in start
self._popen = self._Popen(self)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 281, in _Popen
return Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 19, in init
self._launch(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap
self.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 360, in step2
self.post_quantization_optimization()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 422, in post_quantization_optimization
self._finetune()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 712, in _finetune
finetune.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 54, in run
return super().run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 150, in run
self._run_int()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 387, in _run_int
self.run_qft(self._model_native, self._model, metrics=self.metrics)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 507, in run_qft
self.main_train_summary_per_epoch = qft_distiller.fit(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1685, in fit
tmp_logs = self.train_function(iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1284, in train_function
return step_function(self, iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1268, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1250, in _distributed_apply_gradients_fn
distribution.extended.update(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1245, in apply_grad_to_update_var
return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: ‘update_1_1/StatefulPartitionedCall’
Detected at node ‘update_1_1/StatefulPartitionedCall’ defined at (most recent call last):
File “/home/admin/dani/4_optimization.py”, line 21, in
runner.optimize(calib_dataset_path)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2128, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1970, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1125, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1319, in _full_acceleras_run
optimization_flow.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 335, in run
step_func()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 94, in parent_wrapper
proc.start()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 121, in start
self._popen = self._Popen(self)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 281, in _Popen
return Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 19, in init
self._launch(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap
self.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 360, in step2
self.post_quantization_optimization()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 422, in post_quantization_optimization
self._finetune()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 712, in _finetune
finetune.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 54, in run
return super().run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 150, in run
self._run_int()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 387, in _run_int
self.run_qft(self._model_native, self._model, metrics=self.metrics)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 507, in run_qft
self.main_train_summary_per_epoch = qft_distiller.fit(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1685, in fit
tmp_logs = self.train_function(iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1284, in train_function
return step_function(self, iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1268, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1250, in _distributed_apply_gradients_fn
distribution.extended.update(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1245, in apply_grad_to_update_var
return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: ‘update_1_1/StatefulPartitionedCall’
Detected at node ‘update_1_1/StatefulPartitionedCall’ defined at (most recent call last):
File “/home/admin/dani/4_optimization.py”, line 21, in
runner.optimize(calib_dataset_path)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2128, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1970, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1125, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1319, in _full_acceleras_run
optimization_flow.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 335, in run
step_func()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 94, in parent_wrapper
proc.start()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 121, in start
self._popen = self._Popen(self)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/context.py”, line 281, in _Popen
return Popen(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 19, in init
self._launch(process_obj)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/popen_fork.py”, line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap
self.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 360, in step2
self.post_quantization_optimization()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 422, in post_quantization_optimization
self._finetune()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 712, in _finetune
finetune.run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 54, in run
return super().run()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 150, in run
self._run_int()
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 387, in _run_int
self.run_qft(self._model_native, self._model, metrics=self.metrics)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/hailo_model_optimization/algorithms/finetune/qft.py”, line 507, in run_qft
self.main_train_summary_per_epoch = qft_distiller.fit(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1685, in fit
tmp_logs = self.train_function(iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1284, in train_function
return step_function(self, iterator)
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/engine/training.py”, line 1268, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1250, in _distributed_apply_gradients_fn
distribution.extended.update(
File “/home/admin/anaconda3/envs/conda_env/lib/python3.10/site-packages/keras/optimizers/optimizer.py”, line 1245, in apply_grad_to_update_var
return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: ‘update_1_1/StatefulPartitionedCall’
3 root error(s) found.
(0) INTERNAL: libdevice not found at ./libdevice.10.bc
[[{{node update_1_1/StatefulPartitionedCall}}]]
[[update_2_144/AssignAddVariableOp/_2781]]
[[group_deps/_2789]]
(1) INTERNAL: libdevice not found at ./libdevice.10.bc
[[{{node update_1_1/StatefulPartitionedCall}}]]
[[update_2_144/AssignAddVariableOp/_2781]]
(2) INTERNAL: libdevice not found at ./libdevice.10.bc
[[{{node update_1_1/StatefulPartitionedCall}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_582293]

4

I solved the problem, and DFC is now working in my Conda environment.
I exported the following environment variables:

export CUDA_HOME="/home/admin/anaconda3/envs/conda_env/nvvm"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/admin/anaconda3/envs/conda_env/nvvm/libdevice"
export XLA_FLAGS="--xla_gpu_cuda_data_dir=/home/admin/anaconda3/envs/conda_env"

Change the paths according to the location of your nvvm folder.
I suggest modifying your ~/.bashrc to save these variables permanently. After editing, run source ~/.bashrc to apply the changes.