DFC/MZ - Optimization failure with DETR

Hey!

In the past couple of days, I’ve been playing around with Hailo DFC (3.29.0), trying to convert YoloV11 and DETR. I’ve been using DFC directly and then switched to hailomz to make sure I’m not messing anything up, since hailomz already has all the relevant configs.

Unfortunately, I have never managed to actually get to the .hef format from the source .onnx files.

I’ve searched around the forum and found the same error in another thread (DFC 3.29 - Failing on optimize), but the thread has been inactive for a bit, so I decided to create my own.

Relevant system info

  1. CPU - i9-9900k, RAM - 96GB DDR4, GPU - 2xGTX 1080
  2. Ubuntu 22.04, Kernel - 6.8.0-40-generic

What I’ve done to get to this point

  1. Installed hailo dfc 3.29.0
  2. Installed hailomz (commit #cef737ba)
  3. Downloaded the calibration dataset with python hailo_model_zoo/datasets/create_coco_tfrecord.py calib2017
  4. Ran the optimisation with hailomz optimize detr_resnet_v1_50 --calib-path ~/.hailomz/data/models_files/coco/2023-08-03/coco_calib2017.tfrecord

The error

$ hailomz optimize detr_resnet_v1_50 --calib-path ~/.hailomz/data/models_files/coco/2023-08-03/coco_calib2017.tfrecord
<Hailo Model Zoo INFO> Start run for network detr_resnet_v1_50 ...
<Hailo Model Zoo INFO> Initializing the hailo8 runner...
[info] Translation started on ONNX model detr_resnet_v1_50
[info] Restored ONNX model detr_resnet_v1_50 (completion time: 00:00:00.61)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:01.98)
[info] Start nodes mapped from original model: 'test_input': 'detr_resnet_v1_50/input_layer1'.
[info] End nodes mapped from original model: '/class_embed/Add', '/Sigmoid'.
[info] Translation completed on ONNX model detr_resnet_v1_50 (completion time: 00:00:06.16)
[info] Saved HAR to: /home/xwitpe00/hailo_model_zoo/detr_resnet_v1_50.har
<Hailo Model Zoo INFO> Preparing calibration data...
[info] Loading model script commands to detr_resnet_v1_50 from /home/xwitpe00/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/detr_resnet_v1_50.alls
[info] Starting Model Optimization
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:02.24)
[info] Starting LayerNorm Decomposition
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████| 64/64 [02:30<00:00,  2.35s/entries]
[info] LayerNorm Decomposition is done (completion time is 00:02:56.23)
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████| 64/64 [07:28<00:00,  7.01s/entries]
[info] Statistics Collector is done (completion time is 00:07:46.66)
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.09)
[info] Starting Matmul Equalization
Traceback (most recent call last):
  File "/home/xwitpe00/hailo/hailo/bin/hailomz", line 33, in <module>
    sys.exit(load_entry_point('hailo-model-zoo', 'console_scripts', 'hailomz')())
  File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main.py", line 122, in main
    run(args)
  File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main.py", line 111, in run
    return handlers[args.command](args)
  File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/main_driver.py", line 227, in optimize
    optimize_model(
  File "/home/xwitpe00/hailo_model_zoo/hailo_model_zoo/core/main_utils.py", line 326, in optimize_model
    runner.optimize(calib_feed_callback)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2093, in optimize
    self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1935, in _optimize
    self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1045, in full_quantization
    self._full_acceleras_run(self.calibration_data, data_type)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1229, in _full_acceleras_run
    optimization_flow.run()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 306, in wrapper
    return func(self, *args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 326, in run
    step_func()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 111, in parent_wrapper
    raise SubprocessTracebackFailure(*child_messages)
hailo_model_optimization.acceleras.utils.acceleras_exceptions.SubprocessTracebackFailure: Subprocess failed with traceback

Traceback (most recent call last):
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 73, in child_wrapper
    func(self, *args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 345, in step1
    self.core_quantization()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 402, in core_quantization
    self._matmul_equalization()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 582, in _matmul_equalization
    algo.run()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 50, in run
    return super().run()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 150, in run
    self._run_int()
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 123, in _run_int
    self.equalize_input_paths(normal_subflow, transpose_subflow, layer)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 154, in equalize_input_paths
    set_scales_forward(self._model, normal_flow, layer.groups, factors)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 332, in set_scales_forward
    scales, zp_points = get_scales_output(in_layer, groups, factors, transpose)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/matmul_equalization.py", line 359, in get_scales_output
    scales, zp = find_the_best_zp(
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/matmul_equalization/optimal_zp_finder.py", line 237, in find_the_best_zp
    result = minimize(objective, x0, method="SLSQP", constraints=cons)
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/scipy/optimize/_minimize.py", line 722, in minimize
    res = _minimize_slsqp(fun, x0, args, jac, bounds,
  File "/home/xwitpe00/hailo/hailo/lib/python3.10/site-packages/scipy/optimize/_slsqp_py.py", line 431, in _minimize_slsqp
    slsqp(m, meq, x, xl, xu, fx, c, g, a, acc, majiter, mode, w, jw,
ValueError: failed to initialize intent(inout) array -- expected elsize=8 but got 4

Hi @lukasczk,
It seems to be a bug in our SW. Thank you for pointing this out. I’ll take that to our R&D.

Regards,

1 Like

Hi @lukasczk,
To solve the “ValueError: failed to initialize intent(inout) array – expected elsize=8 but got 4”, please downgrade the scipy version to 1.9.3 (pip install scipy==1.9.3)

Regards,

1 Like

Hey, thanks, will try it out!