Hi when running hailomz optimize yolov8s_pose on my system I get the following error:
Start run for network yolov8s_pose …
Initializing the hailo8 runner…
[info] Translation started on ONNX model yolov8s_pose
[info] Restored ONNX model yolov8s_pose (completion time: 00:00:00.19)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.62)
[info] Start nodes mapped from original model: ‘images’: ‘yolov8s_pose/input_layer1’.
[info] End nodes mapped from original model: ‘/model.22/cv2.2/cv2.2.2/Conv’, ‘/model.22/cv3.2/cv3.2.2/Conv’, ‘/model.22/cv4.2/cv4.2.2/Conv’, ‘/model.22/cv2.1/cv2.1.2/Conv’, ‘/model.22/cv3.1/cv3.1.2/Conv’, ‘/model.22/cv4.1/cv4.1.2/Conv’, ‘/model.22/cv2.0/cv2.0.2/Conv’, ‘/model.22/cv3.0/cv3.0.2/Conv’, ‘/model.22/cv4.0/cv4.0.2/Conv’.
[info] Translation completed on ONNX model yolov8s_pose (completion time: 00:00:01.17)
[info] Saved HAR to: /data2/neoklisv/hailo_model_zoo/yolov8s_pose.har
Preparing calibration data…
[info] Loading model script commands to yolov8s_pose from /data2/neoklisv/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s_pose.alls
[info] Starting Model Optimization
[info] Using default optimization level of 2
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:00.56)
[info] Layer Norm Decomposition skipped
[info] Starting Stats Collector
[info] Using dataset with 64 entries for calibration
Calibration: 0%| | 0/64 [00:28<?, ?entries/s]
Traceback (most recent call last):
File “/home/neoklisv/miniconda3/envs/hailo_cuda/bin/hailomz”, line 33, in
sys.exit(load_entry_point(‘hailo-model-zoo’, ‘console_scripts’, ‘hailomz’)())
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main.py”, line 122, in main
run(args)
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main.py”, line 111, in run
return handlersargs.command
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main_driver.py”, line 227, in optimize
optimize_model(
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/core/main_utils.py”, line 321, in optimize_model
runner.optimize(calib_feed_callback)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2093, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1935, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1045, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1229, in _full_acceleras_run
optimization_flow.run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 316, in run
step_func()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 111, in parent_wrapper
raise SubprocessTracebackFailure(*child_messages)
hailo_model_optimization.acceleras.utils.acceleras_exceptions.SubprocessTracebackFailure: Subprocess failed with traceback
Traceback (most recent call last):
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 334, in step1
self.pre_quantization_optimization()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 374, in pre_quantization_optimization
self._collect_stats()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 449, in _collect_stats
stats_collector.run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 50, in run
return super().run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 151, in run
self._run_int()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/stats_collection/stats_collection.py”, line 38, in _run_int
stats_collector.collect_stats(
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/statistics/stats_collector.py”, line 111, in collect_stats
acceleras_model.predict_on_batch(preprocessed_data)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2603, in predict_on_batch
outputs = self.predict_function(iterator)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py”, line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/tensorflow/python/eager/execute.py”, line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
Detected at node ‘hailo_model/conv1/conv_op/Conv2D’ defined at (most recent call last):
File “/home/neoklisv/miniconda3/envs/hailo_cuda/bin/hailomz”, line 33, in
sys.exit(load_entry_point(‘hailo-model-zoo’, ‘console_scripts’, ‘hailomz’)())
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main.py”, line 122, in main
run(args)
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main.py”, line 111, in run
return handlersargs.command
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/main_driver.py”, line 227, in optimize
optimize_model(
File “/data2/neoklisv/hailo_model_zoo/hailo_model_zoo/core/main_utils.py”, line 321, in optimize_model
runner.optimize(calib_feed_callback)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2093, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1935, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1045, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1229, in _full_acceleras_run
optimization_flow.run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 316, in run
step_func()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 94, in parent_wrapper
proc.start()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/process.py”, line 121, in start
self._popen = self._Popen(self)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/context.py”, line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/context.py”, line 277, in _Popen
return Popen(process_obj)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/popen_fork.py”, line 19, in init
self._launch(process_obj)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/popen_fork.py”, line 75, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 73, in child_wrapper
func(self, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 334, in step1
self.pre_quantization_optimization()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 374, in pre_quantization_optimization
self._collect_stats()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 449, in _collect_stats
stats_collector.run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py”, line 50, in run
return super().run()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/algorithm_base.py”, line 151, in run
self._run_int()
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/algorithms/stats_collection/stats_collection.py”, line 38, in _run_int
stats_collector.collect_stats(
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/statistics/stats_collector.py”, line 111, in collect_stats
acceleras_model.predict_on_batch(preprocessed_data)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2603, in predict_on_batch
outputs = self.predict_function(iterator)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2169, in predict_function
return step_function(self, iterator)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2155, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2143, in run_step
outputs = model.predict_step(data)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 2111, in predict_step
return self(x, training=False)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/training.py”, line 558, in call
return super().call(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/base_layer.py”, line 1145, in call
outputs = call_fn(inputs, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1118, in call
for lname in self.flow.toposort():
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1125, in call
output = self._call_layer(
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1251, in _call_layer
outputs = acceleras_layer(inputs, training=training, encoding_tensors=encoding_tensors, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/base_layer.py”, line 1145, in call
outputs = call_fn(inputs, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 148, in call
for op_name in self._layer_flow.toposort_ops():
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 159, in call
op_result = op(
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 65, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/engine/base_layer.py”, line 1145, in call
outputs = call_fn(inputs, *args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 96, in error_handler
return fn(*args, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/atomic_ops/base_atomic_op.py”, line 1029, in call
outputs = self._native_run(inputs, training=training, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/atomic_ops/base_atomic_op.py”, line 1121, in _native_run
outputs = self.call_native(native_input, training=training, **kwargs)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/atomic_ops/conv_stripped_op.py”, line 867, in call_native
return self._call_conv_internal(inputs[0], self.kernel, self.padding_const_value)
File “/home/neoklisv/miniconda3/envs/hailo_cuda/lib/python3.8/site-packages/hailo_model_optimization/acceleras/atomic_ops/conv_stripped_op.py”, line 1082, in _call_conv_internal
output = tf.nn.conv2d(
Node: ‘hailo_model/conv1/conv_op/Conv2D’
DNN library is not found.
[[{{node hailo_model/conv1/conv_op/Conv2D}}]] [Op:__inference_predict_function_85343]
Initially I used the system-wide nvcc tools which was 12.5, then using conda I switched to 11.8 without any luck. Can you help me? (Ubuntu 20.04, Python 3.8)