Regarding the issue of converting ONNX to HEF

General
1

I have two ONNX models, both trained using the same code. The only difference is in the loss calculation method; all other parameters remain unchanged.

I also used the same code to convert them into ONNX format. However, only one of the ONNX models can be successfully converted to HEF, while the other fails. Below is the error message I received.

Does anyone know what might be causing this?

(hailo) joy@Honestmc-Linux:~/Desktop/Hailo_Project_Final/Guidewire_Seg$ ./run_onnx2hef.sh --onnx weights/20250717/best_simplified.onnx calib_data/calib_data_1024.npy weights/20250717/unet_norm.alls 
่ผธๅ…ฅๆช”ๆกˆๆ ผๅผๆญฃ็ขบ
ๆช”ๅ: best_simplified
ๅ‰ฏๆช”ๅ: onnx
่ณ‡ๆ–™ๅคพ่ทฏๅพ‘๏ผšweights/20250717
---- ่ฝ‰ๆ›(parse) onnx to har ----
[info] Current Time: 11:49:51, 07/17/25
[info] CPU: Architecture: x86_64, Model: AMD Ryzen 7 5700G with Radeon Graphics, Number Of Cores: 16, Utilization: 0.3%
[info] Memory: Total: 14GB, Available: 10GB
[info] System info: OS: Linux, Kernel: 6.8.0-59-generic
[info] Hailo DFC Version: 3.29.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo parser onnx weights/20250717/best_simplified.onnx --hw-arch hailo8 --har-path weights/20250717/best_simplified_hailo8_parse.har -y`
[info] Translation started on ONNX model best_simplified
[info] Restored ONNX model best_simplified (completion time: 00:00:00.07)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.21)
[info] Start nodes mapped from original model: 'input_2': 'best_simplified/input_layer1'.
[info] End nodes mapped from original model: '/segmentation_head/segmentation_head.2/activation/Sigmoid'.
[info] Translation completed on ONNX model best_simplified (completion time: 00:00:00.48)
[info] Saved HAR to: /home/joy/Desktop/Hailo_Project_Final/Guidewire_Seg/weights/20250717/best_simplified_hailo8_parse.har
---- ๆœ€ไฝณๅŒ–(optimize) har ----
[info] Current Time: 11:49:58, 07/17/25
[info] CPU: Architecture: x86_64, Model: AMD Ryzen 7 5700G with Radeon Graphics, Number Of Cores: 16, Utilization: 0.4%
[info] Memory: Total: 14GB, Available: 10GB
[info] System info: OS: Linux, Kernel: 6.8.0-59-generic
[info] Hailo DFC Version: 3.29.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo optimize weights/20250717/best_simplified_hailo8_parse.har --hw-arch hailo8 --calib-set-path calib_data/calib_data_1024.npy --output-har-path weights/20250717/best_simplified_hailo8_optimized.har --work-dir weights/20250717/compile_msg_hailo8 --model-script weights/20250717/unet_norm.alls`
[info] Loading model script commands to best_simplified from weights/20250717/unet_norm.alls
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won't be optimized and compression won't be used) because there's no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] Starting Mixed Precision
[info] Mixed Precision is done (completion time is 00:00:00.35)
[info] Remove layer best_simplified/conv19 because it has no effect on the network output
[info] Remove layer best_simplified/dw10 because it has no effect on the network output
[info] Remove layer best_simplified/conv20 because it has no effect on the network output
[info] Remove layer best_simplified/ew_add6 because it has no effect on the network output
[info] LayerNorm Decomposition skipped
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 64/64 [00:20<00:00,  3.19entries/s]
[info] Statistics Collector is done (completion time is 00:00:20.98)
[info] Output layer best_simplified/conv46 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [5.038887859559241e-34, 1.0]).
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] No shifts available for layer best_simplified/conv37/conv_op, using max shift instead. delta=2.6969
[info] No shifts available for layer best_simplified/conv37/conv_op, using max shift instead. delta=1.3484
Traceback (most recent call last):
  File "/home/joy/miniconda3/envs/hailo/bin/hailo", line 8, in <module>
    sys.exit(main())
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/main.py", line 111, in main
    ret_val = client_command_runner.run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 68, in run
    return self._run(argv)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 89, in _run
    return args.func(args)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/optimize_cli.py", line 120, in run
    self._runner.optimize(dataset, work_dir=args.work_dir)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2093, in optimize
    self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1935, in _optimize
    self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1045, in full_quantization
    self._full_acceleras_run(self.calibration_data, data_type)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1229, in _full_acceleras_run
    optimization_flow.run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 306, in wrapper
    return func(self, *args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 326, in run
    step_func()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 124, in parent_wrapper
    func(self, *args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 345, in step1
    self.core_quantization()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 403, in core_quantization
    self._create_hw_params()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py", line 250, in wrapped
    result = method(*args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 451, in _create_hw_params
    create_hw_params.run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 50, in run
    return super().run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 150, in run
    self._run_int()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 340, in _run_int
    comp_to_retry = self._create_hw_params_component(matching_component_group)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 210, in _create_hw_params_component
    retry_negative_exp_list = self._hanlde_negative_exponent(layer, matching_component_group)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 226, in _hanlde_negative_exponent
    algo.run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 50, in run
    return super().run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 150, in run
    self._run_int()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 77, in _run_int
    l_fix = self.fix_output(l_fix)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 123, in fix_output
    self._log_negative_exponent_shift(l_fix)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 227, in _log_negative_exponent_shift
    raise NegativeSlopeExponentNonFixable(
hailo_model_optimization.acceleras.utils.acceleras_exceptions.NegativeSlopeExponentNonFixable: Quantization failed in layer best_simplified/conv15 due to unsupported required slope. Desired shift is 11.0, but op has only 8 data bits. This error raises when the data or weight range are not balanced. Mostly happens when using random calibration-set/weights, the calibration-set is not normalized properly or batch-normalization was not used during training.
---- ็ทจ่ญฏ(compiler) har to hef----
[info] Current Time: 11:50:37, 07/17/25
[info] CPU: Architecture: x86_64, Model: AMD Ryzen 7 5700G with Radeon Graphics, Number Of Cores: 16, Utilization: 0.4%
[info] Memory: Total: 14GB, Available: 10GB
[info] System info: OS: Linux, Kernel: 6.8.0-59-generic
[info] Hailo DFC Version: 3.29.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler weights/20250717/best_simplified_hailo8_optimized.har --hw-arch hailo8 --output-dir weights/20250717`
Traceback (most recent call last):
  File "/home/joy/miniconda3/envs/hailo/bin/hailo", line 8, in <module>
    sys.exit(main())
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/main.py", line 111, in main
    ret_val = client_command_runner.run()
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 68, in run
    return self._run(argv)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 89, in _run
    return args.func(args)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/compiler_cli.py", line 52, in run
    self._compile(
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/compiler_cli.py", line 72, in _compile
    runner = self._initialize_runner(model_path, quantized_weights_path, hw_arch)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/compiler_cli.py", line 108, in _initialize_runner
    runner = ClientRunner(har=model_path, hw_arch=hw_arch)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 139, in __init__
    self.load_har(har=har)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1850, in load_har
    har = HailoArchive.load(har_path, temp_dir=self._temp_dir.name)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/hailo_archive/hailo_archive.py", line 499, in load
    with HailoArchiveLoader(har_path) as har_loader:
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/hailo_archive/hailo_archive.py", line 508, in __init__
    self._har_file = tarfile.open(self._path, "r:*")
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/tarfile.py", line 1833, in open
    return func(name, "r", fileobj, **kwargs)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/tarfile.py", line 1899, in gzopen
    fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/home/joy/miniconda3/envs/hailo/lib/python3.10/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'weights/20250717/best_simplified_hailo8_optimized.har'
mv: ็„กๆณ•ๅ–ๅพ— 'weights/20250717/best_simplified.hef' ็š„่ณ‡่จŠ: ๆฒ’ๆœ‰ๆญคไธ€ๆช”ๆกˆๆˆ–็›ฎ้Œ„
็ทจ่ญฏๅฎŒๆˆ๏ผŒ่ผธๅ‡บๆช”ๆกˆ๏ผšweights/20250717/best_simplified_hailo8.hef

And this is my ONNXtoHef .sh

#!/bin/bash
# $ ./run_onnx2hef.sh --onnx weights/20250421/best_simplified.onnx calib_data/calib_data_300.npy weights/20250624/unet_norm.alls
if [ "$#" -lt 2 ]; then
    echo "Usage: $0 --onnx filename.onnx"
    exit 1
fi

if [ "$1" != "--onnx" ]; then
    echo "The first argument must be --onnx"
    exit 1
fi

file="$2"

if [[ "$file" == *.onnx ]]; then
    echo "Input file format is correct"
else
    echo "Provided file is not in .onnx format"
    exit 1
fi

# If you are using Miniconda installed in the default location ~/miniconda3
if [ -f "$HOME/miniconda3/etc/profile.d/conda.sh" ]; then
    source "$HOME/miniconda3/etc/profile.d/conda.sh"
else
    echo "Cannot find conda initialization script"
    exit 1
fi

hailo_model_name="hailo8"

# Extract filename and extension from parameter
filename=$(basename "$file")
name="${filename%%.*}"
extension="${filename##*.}"
output_folder=$(dirname "$file")
msg_folder="$output_folder"/compile_msg_"$hailo_model_name"

echo "Filename: $name"
echo "Extension: $extension"
echo "Directory path: $output_folder"

mkdir -p "$msg_folder"

parse_file="$output_folder"/"$name"_"$hailo_model_name"_parse.har
optimize_file="$output_folder"/"$name"_"$hailo_model_name"_optimized.har
compile_file="$output_folder"/"$name".hef
compile_file_har="$output_folder"/"$name"_"$hailo_model_name"_compiled.har

conda activate hailo

echo "---- Parsing ONNX to HAR ----"
hailo parser onnx "$file" --hw-arch "$hailo_model_name" --har-path "$parse_file" -y

echo "---- Optimizing HAR ----"
hailo optimize "$parse_file" --hw-arch "$hailo_model_name" --calib-set-path "$3" --output-har-path "$optimize_file" --work-dir "$msg_folder" --model-script "$4"
# hailo optimize "$parse_file" --hw-arch "$hailo_model_name" --calib-set-path "$3" --model-script unet_mobilenet_v2.alls --output-har-path "$optimize_file"
# hailo optimize "$parse_file" --hw-arch "$hailo_model_name" --use-random-calib-set --output-har-path "$optimize_file" --model-script "$4" --quantization-threshold 0.01

echo "---- Compiling HAR to HEF ----"
hailo compiler "$optimize_file" --hw-arch "$hailo_model_name" --output-dir "$output_folder"
mv "$compile_file" "$output_folder"/"$name"_"$hailo_model_name".hef
echo "Compilation completed. Output file: $output_folder/$name"_"$hailo_model_name.hef"

And this is my .alls file

normalization1 = normalization([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
2

Hey @Yen ,

In the optimize phase of the DFC workflow, the quantizer tried to assign an 11-bit shift to the conv15 layerโ€”yet the Hailo-8 hardware only allows up to an 8-bit exponent. In simpler terms, the model needed more precision than the chip can handle.

This typically happens when:

  • Calibration data doesnโ€™t match training data.
    Using random or poorly normalized images during calibration inflates the observed activation ranges, forcing the quantizer to ask for a larger shift.

  • Activations or weights are unbalanced.
    Dropping BatchNorm (or other normalization) in training can let some layersโ€™ outputs swing wildly, again requiring more than 8 bits to represent them.

Because you only tweaked the loss function between runs, the newer model probably ended up with broader or shifted activation/weight distributionsโ€”and the default calibration then hit that 11-bit wall.

1. Use a True, Representative Calibration Set

Swap out any toy or random images for real, properly preprocessed samples from your training domain:.

Make sure your .alls script mirrors exactly how you normalized inputs during training. This keeps your activations within their expected ranges.

2. Re-introduce (or Fuse) BatchNorm

If you removed BatchNorm or other per-channel scaling layers during training, put them backโ€”or run a quantization-aware training pass. That way, each convolution already sees normalized inputs, preventing โ€œrogueโ€ values that blow out your bit budget.

Hope this helps!

1 Like