Hey!
In the past couple of days, I’ve been playing around with Hailo DFC (3.29.0), trying to convert YoloV11 and DETR. I’ve been using DFC directly and then switched to hailomz to make sure I’m not messing anything up, since hailomz already has all the relevant configs.
Unfortunately, I have never managed to actually get to the .hef format from the source .onnx files.
I’ve searched around the forum and found the same error in another thread (DFC 3.29 - Failing on optimize), but the thread has been inactive for a bit, so I decided to create my own.
Relevant system info
- CPU - i9-9900k, RAM - 96GB DDR4, GPU - 2xGTX 1080
- Ubuntu 22.04, Kernel - 6.8.0-40-generic
What I’ve done to get to this point
- Installed hailo dfc 3.29.0
- Installed hailomz (commit #cef737ba)
- Downloaded the calibration dataset with
python hailo_model_zoo/datasets/create_coco_tfrecord.py calib2017
- Ran the optimisation with
hailomz optimize detr_resnet_v1_50 --calib-path ~/.hailomz/data/models_files/coco/2023-08-03/coco_calib2017.tfrecord
The error
$ hailomz optimize detr_resnet_v1_50 --calib-path ~/.hailomz/data/models_files/coco/2023-08-03/coco_calib2017.tfrecord
<Hailo Model Zoo INFO> Start run for network detr_resnet_v1_50 ...
<Hailo Model Zoo INFO> Initializing the hailo8 runner...
[info] Translation started on ONNX model detr_resnet_v1_50
[info] Restored ONNX model detr_resnet_v1_50 (completion time: 00:00:00.61)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:01.98)
[info] Start nodes mapped from original model: 'test_input': 'detr_resnet_v1_50/input_layer1'.
[info] End nodes mapped from original model: '/class_embed/Add', '/Sigmoid'.
[info] Translation completed on ONNX model detr_resnet_v1_50 (completion time: 00:00:06.16)
[info] Saved HAR to: /home/xwitpe00/hailo_model_zoo/detr_resnet_v1_50.har
<Hailo Model Zoo INFO> Preparing calibration data...
[info] Loading model script commands to detr_resnet_v1_50 from /home/xwitpe00/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/detr_resnet_v1_50.alls
[info] Starting Model Optimization
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:02.24)
[info] Starting LayerNorm Decomposition
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████| 64/64 [02:30<00:00, 2.35s/entries]
[info] LayerNorm Decomposition is done (completion time is 00:02:56.23)
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████| 64/64 [07:28<00:00, 7.01s/entries]
[info] Statistics Collector is done (completion time is 00:07:46.66)
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.09)
[info] Starting Matmul Equalization
Traceback (most recent call last):
File "/home/xwitpe00/hailo/hailo/bin/hailomz", line 33, in <module>
sys.exit(load_entry_point('hailo-model-zoo', 'console_scripts', 'hailomz')())
File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main.py", line 122, in main
run(args)
File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main.py", line 111, in run
return handlers[args.command](args)
File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main_driver.py", line 227, in optimize
optimize_model(
File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/core/main_utils.py", line 326, in optimize_model
runner.optimize(calib_feed_callback)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
return func(self, *args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2093, in optimize
self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
return func(self, *args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1935, in _optimize
self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1045, in full_quantization
self._full_acceleras_run(self.calibration_data, data_type)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1229, in _full_acceleras_run
optimization_flow.run()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 306, in wrapper
return func(self, *args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 326, in run
step_func()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
result = method(*args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 111, in parent_wrapper
raise SubprocessTracebackFailure(*child_messages)
hailo_model_optimization.acceleras.utils.acceleras_exceptions.SubprocessTracebackFailure: Subprocess failed with traceback
Traceback (most recent call last):
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 73, in child_wrapper
func(self, *args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 345, in step1
self.core_quantization()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
result = method(*args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 402, in core_quantization
self._matmul_equalization()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
result = method(*args, **kwargs)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 582, in _matmul_equalization
algo.run()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 50, in run
return super().run()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 150, in run
self._run_int()
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 123, in _run_int
self.equalize_input_paths(normal_subflow, transpose_subflow, layer)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 154, in equalize_input_paths
set_scales_forward(self._model, normal_flow, layer.groups, factors)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 332, in set_scales_forward
scales, zp_points = get_scales_output(in_layer, groups, factors, transpose)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 359, in get_scales_output
scales, zp = find_the_best_zp(
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/optimal_zp_finder.py", line 237, in find_the_best_zp
result = minimize(objective, x0, method="SLSQP", constraints=cons)
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/scipy/optimize/_minimize.py", line 722, in minimize
res = _minimize_slsqp(fun, x0, args, jac, bounds,
File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/scipy/optimize/_slsqp_py.py", line 431, in _minimize_slsqp
slsqp(m, meq, x, xl, xu, fx, c, g, a, acc, majiter, mode, w, jw,
ValueError: failed to initialize intent(inout) array -- expected elsize=8 but got 4