Failing on Model_Optimization

General
1

DFC_2_Model_Optimization_Tutorial

Error while trying to runner.optimize(calib_dataset_lq) entered the code and the following error occurred.

File /local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/model_modifications_commands.py:296, in NormalizationCommand.validate_command(self, layer_scope_from_hn)
292 invalid_layer_names = [
293 layer_name for layer_name in self._normalization_layers if layer_name in layer_scope_from_hn
294 ]
295 if invalid_layer_names:
→ 296 raise AllocatorScriptParserException(
297 f"Given layer names {invalid_layer_names} exist in the model. Please use different names",
298 )

AllocatorScriptParserException: Given layer names [‘ERNet_M3_02-2_LOL_v1/normalization1’] exist in the model. Please use different names

How should I fix it…

The runner.load_model_script (all) is below.
‘ERNet_M3_02-2_LOL_v1/Normalization1’ is included

[info] Loading model script commands to ERNet_M3_02-2_LOL_v1 from string
OrderedDict([(‘name’, ‘ERNet_M3_02-2_LOL_v1’),
(‘net_params’,
OrderedDict([(‘version’, ‘1.0’),
(‘stage’, ‘HN’),
(‘clusters_placement’, []),
(‘clusters_to_skip’, ),
(‘output_layers_order’,
[‘ERNet_M3_02-2_LOL_v1/conv22’]),
(‘is_transformer’, False),
(‘transposed_net’, False),
(‘net_scopes’, [‘ERNet_M3_02-2_LOL_v1’]),
(‘lora_adapters’, )])),
(‘layers’,
OrderedDict([(‘ERNet_M3_02-2_LOL_v1/input_layer1’,
OrderedDict([(‘type’, ‘input_layer’),
(‘input’, ),
(‘output’,
[‘ERNet_M3_02-2_LOL_v1/conv1’]),
(‘input_shapes’, [[-1, 256, 256, 3]]),
(‘output_shapes’,
[[-1, 256, 256, 3]]),
(‘original_names’, [‘input’]),
(‘compilation_params’, {}),
(‘quantization_params’, {}),
(‘transposed’, False),
(‘engine’, ‘nn_core’),
(‘io_type’, ‘standard’)])),
(‘ERNet_M3_02-2_LOL_v1/conv1’,
OrderedDict([(‘type’, ‘conv’),
(‘input’,
[‘ERNet_M3_02-2_LOL_v1/input_layer1’]),
(‘output’,
[‘ERNet_M3_02-2_LOL_v1/layer_normalization1’,
‘ERNet_M3_02-2_LOL_v1/ew_add2’]),
(‘input_shapes’, [[-1, 256, 256, 3]]),
(‘output_shapes’,
[[-1, 128, 128, 24],
[-1, 128, 128, 24]]),
(‘original_names’,
[‘/enhancer/embedding/Conv’,
‘/enhancer/encoder_layers.0.0/Transpose’]),
(‘compilation_params’, {}),
(‘quantization_params’, {}),
(‘params’,
OrderedDict([(‘kernel_shape’,
[4, 4, 3, 24]),
(‘strides’,
[1, 2, 2, 1]),
(‘dilations’,
[1, 1, 1, 1]),
(‘padding’,
‘SAME_TENSORFLOW’),
(‘groups’, 1),
(‘layer_disparity’, 1),
(‘input_disparity’, 1),
(‘batch_norm’, False),
(‘elementwise_add’,
False),
(‘activation’,
‘linear’)]))])),
(‘ERNet_M3_02-2_LOL_v1/layer_normalization1’,
OrderedDict([(‘type’, ‘layer_normalization’),
(‘input’,
[‘ERNet_M3_02-2_LOL_v1/conv1’]),
(‘output’,
[‘ERNet_M3_02-2_LOL_v1/normalization1’]),
(‘input_shapes’,
[[-1, 128, 128, 24]]),
(‘output_shapes’,
[[-1, 128, 128, 24]]),
(‘original_names’,
[‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/ReduceMean’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Sub’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Pow’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/ReduceMean_1’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Add’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Sqrt’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Div’]),
(‘compilation_params’, {}),
(‘quantization_params’, {}),
(‘params’,
OrderedDict([(‘reduce_axes’, [3]),
(‘rms_norm’, False),
(‘groups’, 1)]))])),
(‘ERNet_M3_02-2_LOL_v1/normalization1’,
OrderedDict([(‘type’, ‘normalization’),
(‘input’,
[‘ERNet_M3_02-2_LOL_v1/layer_normalization1’]),
(‘output’,
[‘ERNet_M3_02-2_LOL_v1/format_conversion1’]),
(‘input_shapes’,
[[-1, 128, 128, 24]]),
(‘output_shapes’,
[[-1, 128, 128, 24]]),
(‘original_names’,
[‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Mul’,
‘/enhancer/encoder_layers.0.0/blocks.0.0/norm/Add_1’]),
(‘compilation_params’, {}),
(‘quantization_params’, {}),
(‘params’,
OrderedDict([(‘elementwise_add’,
False),
(‘activation’,
‘linear’)]))])),
(‘ERNet_M3_02-2_LOL_v1/format_conversion1’,
OrderedDict([(‘type’, ‘format_conversion’),
(‘input’,
[‘ERNet_M3_02-2_LOL_v1/normalization1’]),
(‘output’,
[‘ERNet_M3_02-2_LOL_v1/conv2’,
‘ERNet_M3_02-2_LOL_v1/conv3’,
‘ERNet_M3_02-2_LOL_v1/conv4’]),
(‘input_shapes’,
[[-1, 128, 128, 24]]),
(‘output_shapes’,
[[-1, 1, 16384, 24],
[-1, 1, 16384, 24],
[-1, 1, 16384, 24]]),
(‘original_names’,
[‘/enhancer/encoder_layers.0.0/blocks.0.0/fn/Reshape’]),
(‘compilation_params’,
{‘hw_layer_type_list’: [‘lcu’]}),
(‘quantization_params’, {}),
(‘params’,
OrderedDict([(‘conversion_type’,
‘spatial_reshape’),
(‘groups’, 1),
(‘spatial_reshape_sizes’,
[1, 16384, 24]),
(‘input_windows’,
[1, 1, 1]),
(‘output_windows’,
[1, 1, 1])]))])),

2

Hi @sslee1,
I think that you’ve took the tutorial template, and transplanted your model into the different commands. I think that in this case, the notebook assumes some layer names. Maybe, start w\o alls at all.

1 Like
3

Thank you.

I’ve tried runner.optimize(calib_dataset_lq) without runner.load_model_script(alls)

and then i’ve encounter another problem…

error message is bellow

[info] Starting Model Optimization [warning] Reducing optimization level to 0 (the accuracy won’t be optimized and compression won’t be used) because there’s less data than the recommended amount (1024), and there’s no available GPU [warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results [info] Model received quantization params from the hn [info] MatmulDecompose skipped [info] Starting Mixed Precision [info] Model Optimization Algorithm Mixed Precision is done (completion time is 00:00:00.35) [info] Starting LayerNorm Decomposition [info] Using dataset with 64 entries for calibration

Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:19<00:00, 3.26entries/s]

[info] Model Optimization Algorithm LayerNorm Decomposition is done (completion time is 00:00:22.37) [info] Starting Statistics Collector [info] Using dataset with 64 entries for calibration

Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [01:04<00:00, 1.00s/entries]

[info] Model Optimization Algorithm Statistics Collector is done (completion time is 00:01:05.56) [info] Starting Fix zp_comp Encoding [info] Model Optimization Algorithm Fix zp_comp Encoding is done (completion time is 00:00:00.01) [info] Starting Matmul Equalization [info] Model Optimization Algorithm Matmul Equalization is done (completion time is 00:00:00.00) [info] activation fitting started for ERNet_M3_02-2_LOL_v1/reduce_sum1/act_op [info] activation fitting started for ERNet_M3_02-2_LOL_v1/activation2/act_op [info] activation fitting started for ERNet_M3_02-2_LOL_v1/reduce_sum2/act_op [info] activation fitting started for ERNet_M3_02-2_LOL_v1/activation4/act_op

…

AccelerasUnsupportedError: layer ERNet_M3_02-2_LOL_v1/matmul1 does not support shift delta. To overcome this issue you should force larger range at the inputs of the layer using command quantization_param([layer_name], force_range_in=[range_min, range_max], force_range_index=index) current range of input 0 is [-0.075, 0.062] and input 1 is [-287933.126, 288390.782].You should increase the multiplication of these ranges by a factor of 7.172, e.g. you can apply factor of sqrt(7.172) to both inputs:
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-0.201, 0.166], force_range_index=0)
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-771101.899, 772327.529], force_range_index=1)

so i tried this link
but from hailo_sdk_client import Client is not work only work from hailo_sdk_client import ClientRunner, InferenceContext

so how can i solve it. the way of load_model_script is also occurred first problem… do i need to leave hailo tutorial notebook? …

4

It seems that now the problem is in the quantization. A few questions-

  1. Are you using real images for the calibration?
  2. Does your images need normalization prior to be fed into the model, if so are you doing it?
5

  1. Are you using real images for the calibration?
    yes. i used real images for the calibration

  2. Does your images need normalization prior to be fed into the model, if so are you doing it?
    yes.

I am currently in the process of converting a low-light enhencement model.

so… i used this function and dataset

First, we will prepare the calibration set. Resize the images to the correct size and crop them.

def preproc(image, output_height=256, output_width=256, resize_side=256):
    """imagenet-standard: aspect-preserving resize to 256px smaller-side, then central-crop to 224px"""
    with eager_mode():
        h, w = image.shape[0], image.shape[1]
        scale = tf.cond(tf.less(h, w), lambda: resize_side / h, lambda: resize_side / w)
        resized_image = tf.compat.v1.image.resize_bilinear(tf.expand_dims(image, 0), [int(h * scale), int(w * scale)])
        cropped_image = tf.compat.v1.image.resize_with_crop_or_pad(resized_image, output_height, output_width)
        cropped_image = cropped_image / 255.0
        return tf.squeeze(cropped_image)


images_path_lq = "./LOLv1/low"
images_path_gt = "./LOLv1/high"

images_list_lq = [img_name for img_name in os.listdir(images_path_lq) if os.path.splitext(img_name)[1] == ".jpg" or os.path.splitext(img_name)[1] == ".png"]
images_list_gt = [img_name for img_name in os.listdir(images_path_gt) if os.path.splitext(img_name)[1] == ".jpg" or os.path.splitext(img_name)[1] == ".png"]


calib_dataset_lq = np.zeros((len(images_list_lq), 256, 256, 3))
for idx, img_name in enumerate(sorted(images_list_lq)):
    img = np.array(Image.open(os.path.join(images_path_lq, img_name)))
    img_preproc = preproc(img)
    calib_dataset_lq[idx, :, :, :] = img_preproc.numpy()

np.save("calib_set_lq.npy", calib_dataset_lq)

calib_dataset_gt = np.zeros((len(images_list_gt), 256, 256, 3))
for idx, img_name in enumerate(sorted(images_list_gt)):
    img = np.array(Image.open(os.path.join(images_path_gt, img_name)))
    img_preproc = preproc(img)
    calib_dataset_gt[idx, :, :, :] = img_preproc.numpy()

np.save("calib_set_gt.npy", calib_dataset_gt)

and

alls = "normalization1 = normalization([123.1, 116.28, 103.53], [58.395, 57.12, 57.375])\n"

not this number … i used real dataset mean and std.

but this command make error runner.load_model_script(alls)

so if i do not above command.
another error are occured.
AccelerasUnsupportedError: layer ERNet_M3_02-2_LOL_v1/matmul1 does not support shift delta. To overcome this issue you should force larger range at the inputs of the layer using command quantization_param([layer_name], force_range_in=[range_min, range_max], force_range_index=index) current range of input 0 is [-0.075, 0.062] and input 1 is [-287933.126, 288390.782].You should increase the multiplication of these ranges by a factor of 7.172, e.g. you can apply factor of sqrt(7.172) to both inputs:
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-0.201, 0.166], force_range_index=0)
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-771101.899, 772327.529], force_range_index=1)

6

Hi,
Why not simply trying what the tool suggested in the error message?

quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-0.201, 0.166], force_range_index=0)
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-771101.899, 772327.529], force_range_index=1)
7

I am working in this environment.

sudo ./hailo_ai_sw_suite_docker_run.sh --resume
version : hailo_ai_sw_suite_2025-01

[info] Current Time: 16:20:03, 02/05/25
[info] CPU: Architecture: x86_64, Model: 13th Gen Intel(R) Core™ i9-13900, Number Of Cores: 32, Utilization: 1.7%
[info] Memory: Total: 46GB, Available: 40GB
[info] System info: OS: Linux, Kernel: 6.8.0-51-generic
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0

I’ve tried this command

quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-0.201, 0.166], force_range_index=0)
quantization_param([ERNet_M3_02-2_LOL_v1/matmul1], force_range_in=[-771101.899, 772327.529], force_range_index=1)

import os
from hailo_sdk_client import ClientRunner

# Model name and HAR file path
original_model_name = "ERNet_M3_02-2_LOL_v1"
hailo_model_har_name = f"{original_model_name}_hailo_model.har"

# Verify HAR file existence
assert os.path.isfile(hailo_model_har_name), "Please provide a valid path for the HAR file"

# Initialize ClientRunner
runner = ClientRunner(har=hailo_model_har_name)

# Quantization parameter setup
# Writing the quantization parameters directly into the script for runner.optimize
quantization_script = f"""
quantization_param(["{original_model_name}/matmul1"], force_range_in=[-0.201, 0.166], force_range_index=0)
quantization_param(["{original_model_name}/matmul1"], force_range_in=[-771101.899, 772327.529], force_range_index=1)
"""

# Write quantization parameters to a script file
model_script_path = "./quantization_script.alls"
with open(model_script_path, "w") as script_file:
    script_file.write(quantization_script)

# Load the model script
try:
    runner.load_model_script(model_script_path)
except Exception as e:
    print(f"Error loading model script: {e}")
    raise

# Prepare calibration dataset (replace with your actual dataset loading code)
calib_dataset_lq = None  # Placeholder: Load your calibration dataset here

# Perform optimization
try:
    runner.optimize(calib_dataset_lq)
except Exception as e:
    print(f"Optimization failed: {e}")
    raise

# Save the optimized HAR file
quantized_model_har_path = f"{original_model_name}_quantized_model.har"
try:
    runner.save_har(quantized_model_har_path)
    print(f"Quantized model saved at: {quantized_model_har_path}")
except Exception as e:
    print(f"Failed to save optimized HAR file: {e}")

this error is occured

File /local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/commands.py:311, in ModelScriptCommand.add_scope_to_layer(scope_names, layer_name, force)
308 if force:
309 return f"{scope_name[0]}/{layer_parts[-1]}"
→ 311 raise AllocatorScriptParserException(f"Invalid scope name {layer_parts[0]} exists")

AllocatorScriptParserException: Invalid scope name "ERNet_M3_02-2_LOL_v1 exists