Optimization/quantization problem

[Nazariy_Vdovenko]

Hello everyone! I have my custom YOLOv8m model, and I want to convert it to .hef format for use on a Raspberry Pi 5. I converted the model to .onnx, then parsed it into .har, but I’m facing issues with optimizing/quantizing the model. I suspect the problem might be due to an incorrect nms_config.json file. Could you help me configure it correctly? At this stage, I’m getting this error:

[info] Loading model script commands to best from string
[info] The activation function of layer best/conv58 was replaced by a Sigmoid
[info] The activation function of layer best/conv71 was replaced by a Sigmoid
[info] The activation function of layer best/conv83 was replaced by a Sigmoid
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won’t be optimized and compression won’t be used) because there’s less data than the recommended amount (1024), and there’s no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
Traceback (most recent call last):
File “/mnt/c/users/usern/desktop/hailo/Hailo8/steps/3_process/optimize.py”, line 69, in
runner.optimize(calib_dataset)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2128, in optimize
self.optimize(calib_data, data_type=data_type, work_dir=work_dir)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 1970, in optimize
self.sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1125, in full_quantization
self.full_acceleras_run(self.calibration_data, data_type)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1319, in full_acceleras_run
optimization_flow.run()
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 306, in wrapper
return func(self, *args, **kwargs)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 335, in run
step_func()
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 123, in parent_wrapper
self.build_model()
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 250, in wrapped
result = method(*args, **kwargs)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 249, in build_model
model.compute_output_shape(shapes)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1091, in compute_output_shape
return self.compute_and_verify_output_shape(input_shape, verify_layer_inputs_shape=False)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1125, in compute_and_verify_output_shape
layer_output_shape = layer.compute_output_shape(layer_input_shapes)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/keras/engine/base_layer.py”, line 917, in compute_output_shape
outputs = self(inputs, training=False)
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/tmp/autograph_generated_filexr2lijvl.py", line 41, in tf__call
outputs = ag.converted_call(ag.ld(self).call_core, (ag.ld(inputs), ag_.ld(training)), dict(**ag__.ld(kwargs)), fscope)
File "/tmp/autograph_generated_filevt28ix_q.py", line 90, in tf__call_core
ag.if_stmt(ag__.ld(self).postprocess_type in [ag__.ld(PostprocessType).NMS, ag__.ld(PostprocessType).BBOX_DECODER], if_body_3, else_body_3, get_state_3, set_state_3, (‘do_return’, ‘retval_’), 2)
File "/tmp/autograph_generated_filevt28ix_q.py", line 22, in if_body_3
retval = ag_.converted_call(ag__.ld(self).bbox_decoding_and_nms_call, (ag__.ld(inputs),), dict(is_bbox_decoding_only=ag__.ld(self).postprocess_type == ag__.ld(PostprocessType).BBOX_DECODER), fscope)
File "/tmp/autograph_generated_filezcykrjfv.py", line 116, in tf__bbox_decoding_and_nms_call
ag.if_stmt(ag__.ld(self).meta_arch in [ag__.ld(NMSOnCpuMetaArchitectures).YOLOV5, ag__.ld(NMSOnCpuMetaArchitectures).YOLOX], if_body_5, else_body_5, get_state_5, set_state_5, (‘decoded_bboxes’, ‘detection_score’, ‘do_return’, ‘retval_’, ‘inputs’), 4)
File "/tmp/autograph_generated_filezcykrjfv.py", line 113, in else_body_5
ag.if_stmt(ag__.ld(self).meta_arch == ag__.ld(NMSOnCpuMetaArchitectures).YOLOV5_SEG, if_body_4, else_body_4, get_state_4, set_state_4, (‘decoded_bboxes’, ‘detection_score’, ‘do_return’, ‘retval_’), 4)
File "/tmp/autograph_generated_filezcykrjfv.py", line 110, in else_body_4
ag.if_stmt(ag__.ld(self).meta_arch == ag__.ld(NMSOnCpuMetaArchitectures).YOLOV8, if_body_3, else_body_3, get_state_3, set_state_3, (‘decoded_bboxes’, ‘detection_score’), 2)
File "/tmp/autograph_generated_filezcykrjfv.py", line 69, in if_body_3
(decoded_bboxes, detection_score) = ag.converted_call(ag__.ld(self).yolov8_decoding_call, (ag__.ld(inputs),), dict(offsets=[0.5, 0.5]), fscope)
File "/tmp/autograph_generated_filesrnr4ai4.py", line 87, in tf__yolov8_decoding_call
decoded_bboxes = ag.converted_call(ag__.ld(tf).expand_dims, (ag__.ld(decoded_bboxes),), dict(axis=2), fscope)
ValueError: Exception encountered when calling layer “yolov8_nms_postprocess” (type HailoPostprocess).

in user code:

File "/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_none_nn_core_layer.py", line 45, in call  *
    outputs = self.call_core(inputs, training, **kwargs)
File "/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_postprocess.py", line 123, in call_core  *
    is_bbox_decoding_only=self.postprocess_type == PostprocessType.BBOX_DECODER,
File "/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_postprocess.py", line 157, in bbox_decoding_and_nms_call  *
    decoded_bboxes, detection_score = self.yolov8_decoding_call(inputs, offsets=[0.5, 0.5])
File "/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_postprocess.py", line 375, in yolov8_decoding_call  *
    decoded_bboxes = tf.expand_dims(decoded_bboxes, axis=2)

ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported.

Call arguments received by layer “yolov8_nms_postprocess” (type HailoPostprocess):
• inputs=[‘tf.Tensor(shape=(None, 80, 80, 64), dtype=float32)’, ‘tf.Tensor(shape=(None, 80, 80, 1), dtype=float32)’, ‘tf.Tensor(shape=(None, 40, 40, 64), dtype=float32)’, ‘tf.Tensor(shape=(None, 40, 40, 1), dtype=float32)’, ‘tf.Tensor(shape=(None, 20, 20, 64), dtype=float32)’, ‘tf.Tensor(shape=(None, 20, 20, 1), dtype=float32)’]
• training=False
• kwargs=<class ‘inspect._empty’>
Exception ignored in: <function ClientRunner.del at 0x7fe935b924d0>
Traceback (most recent call last):
File “/mnt/c/users/usern/desktop/hailo/venv_hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 155, in del
File “/usr/lib/python3.10/tempfile.py”, line 1020, in cleanup
AttributeError: ‘NoneType’ object has no attribute ‘path’

[omria]

Hey @Nazariy_Vdovenko ,

Welcome to the Hailo Community!

** Common Issues in Your Logs**

Your error messages indicate:

• Low optimization level (level 0) due to lack of 1024 calibration images and GPU
• Incorrect or missing Non-Maximum Suppression (NMS) settings
• Possible issue with bounding box decoding due to incorrect nms_config.json values
• Incorrect activation function handling during conversion (conv layers replaced by sigmoid)

---

1. Non-Maximum Suppression (NMS) Configuration

Create a precise nms_config.json with optimized settings:

{
    "iou_th": 0.45,
    "score_th": 0.25,
    "max_det": 300,
    "detections_per_class": 100,
    "nms_type": "greedy",
    "meta_arch": "YOLOV8",
    "use_background_class": false
}

NMS Parameter Breakdown

• iou_th: Intersection over Union threshold (0.45)
• score_th: Confidence score filter (0.25)
• max_det: Maximum detections per image (300)
• nms_type: Greedy suppression for YOLO models
• meta_arch: Specific to YOLOv8 architecture

2. Model Optimization Script

Implement a comprehensive optimization configuration:

runner.load_model_script("""
# Normalization to standardize input
normalization1 = normalization([127.5, 127.5, 127.5], [127.5, 127.5, 127.5])

# Calibration and optimization settings
model_optimization_config(
    calibration, 
    batch_size=8, 
    calibset_size=1024
)

# Activation clipping to improve quantization
pre_quantization_optimization(
    activation_clipping, 
    layers=[*], 
    mode=percentile, 
    clipping_values=[0.01, 99.99]
)

# Weight clipping for model stability
pre_quantization_optimization(
    weights_clipping, 
    layers=[*], 
    mode=percentile, 
    clipping_values=[0.01, 99.99]
)

# Maximum optimization configuration
model_optimization_flavor(
    optimization_level=3, 
    compression_level=0
)
""")

3. Optimization Execution

Run the optimization process:

hailo optimize yolov8.har --calib-path /path/to/calibration_images/

4. Validate Output Nodes

Check model node configuration:

hailomz info yolov8m

Ensure correct node specification:

{
    "start-node-names": ["input"],
    "end-node-names": ["output_1", "output_2", "output_3"]
}