Problem while Model Optimization

General Discussion
,
1

I was trying to compile a custom trained yolov8s model. During that it gives following error.

Traceback (most recent call last):
  File "/local/workspace/optimize.py", line 86, in <module>
    runner.optimize(calibration_data)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2201, in optimize
    result = self._optimize(
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2003, in _optimize
    self._sdk_backend.optimize_full_precision(data_continer)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1722, in optimize_full_precision
    model, params = self._apply_model_modification_commands(model, params, update_model_and_params)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1610, in _apply_model_modification_commands
    model, params = command.apply(model, params, hw_consts=self.hw_arch.consts)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 402, in apply
    self._update_config_file(hailo_nn)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 564, in _update_config_file
    self._update_config_layers(hailo_nn)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 614, in _update_config_layers
    self._set_yolo_config_layers(hailo_nn)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 642, in _set_yolo_config_layers
    conv_layers = self._get_output_preds(hailo_nn)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 624, in _get_output_preds
    raise AllocatorScriptParserException(
hailo_sdk_client.sdk_backend.sdk_backend_exceptions.AllocatorScriptParserException: Error in the last layers of the model, expected conv but found concat layer.

And used .alls and .json file is here.

# .alls
normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
change_output_activation(conv42, linear)
change_output_activation(conv53, linear)
change_output_activation(conv63, linear)
quantization_param([conv42, conv53, conv63], force_range_out=[0.0, 1.0])
nms_postprocess("yolov8s_nms_config.json", meta_arch=yolov8, engine=cpu)


# .json
{
    "nms_scores_th": 0.2,
    "nms_iou_th": 0.7,
    "image_dims": [
        320,
        320
    ],
    "max_proposals_per_class": 100,
    "classes": 1,
    "regression_length": 16,
    "background_removal": false,
    "background_removal_index": 0,
    "bbox_decoders": [
        {
            "name": "bbox_decoder_stride_32",
            "stride": 32,
            "reg_layer": "conv62",
            "cls_layer": "conv63"
        },
        {
            "name": "bbox_decoder_stride_16",
            "stride": 16,
            "reg_layer": "conv52",
            "cls_layer": "conv53"
        },
        {
            "name": "bbox_decoder_stride_8",
            "stride": 8,
            "reg_layer": "conv41",
            "cls_layer": "conv42"
        }
    ]
}

Should i remake onnx file? Or do any other methods?

2

Welcome to the Hailo Community!

I suspect the model was not parsed correctly. Confirm that you chose the correct end-node-names. Compare your parsed model (.har) to the yolov8s model from the ModelZoo. Use the following command to download and parse the model:

hailomz parse yolov8s

You can view the .har files in Netron or use the hailo visualizer command to create a .svg file.

hailo visualizer yolov8s.har

You should have 6 outputs with a conv layer just before each output.

3

Hi @GukHyun_Kim thanks for sharing details. If you want to skip local toolchain setup, the DeGirum Cloud Compiler can compile your Ultralytics YOLO PyTorch checkpoint (.pt) for {{Hailo-8 or Hailo-8L}}). It’s a fast way to validate performance and iterate. It’s currently in early access but you can request access here: https://forms.degirum.com/cloud-compiler

4

Write here your full parse and optamize py codes and I’ll try to help you but I need the full code include how you train the model I want to see the.pt how you made it how you convert to onnx (opset 11 13 or 17 is only OK for hailo) how you parse optamize and compile I’ll help you but provide the code so I’ll have the full Pic. It’s a super duper easy process to make hef file.

5

By the way, engine=cpu - why you don’t run the nms on the hailo chip? Also why you force it?

quantization_param([conv42, conv53, conv63], force_range_out=[0.0, 1.0])

The regretion layer should be bigger 100-200, also maybe you need bias?

Also why not optamize on gpu level 2 (if you have gpu)

It’s not for the parse part but it might help you with optamize to optamize it better…

model_script = ‘’’

=============================================================================

YOLOv8s Optimization Configuration - FIXED VERSION

This configuration addresses the classification layer failure issue

=============================================================================

High-level optimization settings

Level 2 provides the best balance of optimization and accuracy

model_optimization_flavor(optimization_level=2)

=============================================================================

CRITICAL FIX #1: REMOVED force_range_out

=============================================================================

The original script had this problematic line:

quantization_param([conv41, conv42, conv52, conv53, conv62, conv63], force_range_out=[0.0, 1.0])

This was forcing ALL layers to output values between 0-1, which is wrong because:

- Regression layers (conv41, conv52, conv62) need to output pixel coordinates (0-800 range)

- Classification layers (conv42, conv53, conv63) need to output probabilities (0-1 range)

By removing this constraint, each layer can use its natural output range

=============================================================================

Input Normalization - ESSENTIAL for proper quantization

=============================================================================

This tells Hailo how your input images are normalized

YOLOv8 expects images in 0-255 range, then normalized to 0-1 internally

normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])

=============================================================================

Output Activation Functions - CRITICAL for YOLO

=============================================================================

YOLO classification layers MUST use sigmoid activation to output probabilities

These layers output the “confidence” that an object exists at each location

change_output_activation(conv42, sigmoid) # Small object detection head
change_output_activation(conv53, sigmoid) # Medium object detection head
change_output_activation(conv63, sigmoid) # Large object detection head

Note: Regression layers (conv41, conv52, conv62) use linear activation by default

They output bounding box coordinates and don’t need sigmoid

=============================================================================

CRITICAL FIX #2: POST-QUANTIZATION OPTIMIZATION (Bias Correction)

=============================================================================

This is the key fix that will restore your classification layers!

What bias_correction does:

1. Compares original model outputs with quantized model outputs

2. Calculates the systematic error (bias) introduced by quantization

3. Adds correction factors to compensate for this bias

This technique is especially important for YOLO models because:

- Deep networks accumulate quantization errors across many layers

- Sigmoid layers are particularly sensitive to small numerical changes

- Your analysis showed classification layers outputting all zeros - classic bias issue

post_quantization_optimization(bias_correction, policy=enabled)

=============================================================================

Memory Allocation Optimization

=============================================================================

This setting helps with memory efficiency during inference

Keeps it disabled to ensure stable results

allocator_param(width_splitter_defuse=disabled)
‘’’