Mismatching between the number of calibration data and progress bar during ClientRunner.optimize.

General Discussion
,
1

Hello there,

I have worked to optimize PointPillars model (without voxel and middle encoders) with ClientRunner.optimize and 300 calibration data (with a data shape of [300, 496, 432, 64]). While this calibration working without issues, I have a concern with the calibration progress bar, which runs to only 64 that is the last dimension of my calibration data.

My question is this calibration works correctly or it mistook the last dimension as the batch size? Any recommendations to fix this issue? I attached my code and logs as below.

Code:

import os
import glob

import numpy as np
import onnx
from hailo_sdk_client import ClientRunner
# from hailo_sdk_client.exposed_definitions import Dims


if __name__ == '__main__':
    onnx_dir = 'pointpillars_without_mid.onnx'
    hailo_dir = '/local/workspace/pointpillars_without_mid.har'
    calib_dir = 'calib_data1000'
    n_calib = 300

    onnx_dir = os.path.expanduser(onnx_dir)
    hailo_dir = os.path.expanduser(hailo_dir)
    calib_dir = os.path.expanduser(calib_dir)
    onnx.checker.check_model(onnx_dir, full_check=True)

    runner = ClientRunner(hw_arch='hailo8')
    hn, npz = runner.translate_onnx_model(
        onnx_dir,
        'pointpillars_without_mid',
        start_node_names=['x'],
        end_node_names=['cls_score', 'bbox_pred', 'dir_cls_preds'],
        net_input_shapes={'x': [1, 64, 496, 432]},
        # NOTE: not works
        # net_input_format=[Dims.BATCH, Dims.HEIGHT, Dims.WIDTH, Dims.CHANNELS],
    )

    calib_files = glob.glob(f'{calib_dir}/*')

    calib_data = []
    for idx, calib_file in enumerate(calib_files):
        # data: [1, 64, 496, 432]
        data = np.load(calib_file)
        calib_data.append(data)
        if idx >= n_calib + 1:
            break

    # calib_data: [n_calib, 64, 496, 432]
    calib_data = np.concatenate(calib_data, axis=0)
    # calib_data: [n_calib, 496, 432, 64]
    # NOTE: without this transpose will cause shape errors.
    calib_data = np.transpose(calib_data, (0, 2, 3, 1))
    runner.optimize(calib_data)
    runner.save_har(hailo_dir)

Log:

(hailo_virtualenv) hailo@docker-desktop:/local/workspace$ python parse_onnx_pointpillars.py               [info] No GPU chosen and no suitable GPU found, falling back to CPU.
[info] Translation started on ONNX model pointpillars_without_mid
[info] Restored ONNX model pointpillars_without_mid (completion time: 00:00:00.05)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.16)
[info] Start nodes mapped from original model: 'x': 'pointpillars_without_mid/input_layer1'.
[info] End nodes mapped from original model: '/bbox_head/conv_cls/Conv', '/bbox_head/conv_reg/Conv', '/bbox_head/conv_dir_cls/Conv'.
[info] Translation completed on ONNX model pointpillars_without_mid (completion time: 00:00:00.63)
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won't be optimized and compression won't be used) because there's less data than the recommended amount (1024), and there's no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] MatmulDecompose skipped
[info] Starting Mixed Precision
[info] Model Optimization Algorithm Mixed Precision is done (completion time is 00:00:00.06)
[info] LayerNorm Decomposition skipped
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████| 64/64 [02:24<00:00,  2.25s/entries]
[info] Model Optimization Algorithm Statistics Collector is done (completion time is 00:02:24.44)
[info] Starting Fix zp_comp Encoding
[info] Model Optimization Algorithm Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] Starting MatmulDecomposeFix
[info] Model Optimization Algorithm MatmulDecomposeFix is done (completion time is 00:00:00.00)
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.2158
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.1079
[info] No shifts available for layer pointpillars_without_mid/conv16/conv_op, using max shift instead. delta=0.4556
[info] No shifts available for layer pointpillars_without_mid/conv16/conv_op, using max shift instead. delta=0.2278
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.1079
[info] Finetune encoding skipped
[info] Bias Correction skipped
[info] Adaround skipped
[info] Quantization-Aware Fine-Tuning skipped
[info] Layer Noise Analysis skipped
[info] Model Optimization is done
[info] Saved HAR to: /local/workspace/pointpillars_without_mid.har
(hailo_virtualenv) hailo@docker-desktop:/local/workspace$
(hailo_virtualenv) hailo@docker-desktop:/local/workspace$ python parse_onnx_pointpillars.py
[info] No GPU chosen and no suitable GPU found, falling back to CPU.
[info] Translation started on ONNX model pointpillars_without_mid
[info] Restored ONNX model pointpillars_without_mid (completion time: 00:00:00.04)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.16)
[info] Start nodes mapped from original model: 'x': 'pointpillars_without_mid/input_layer1'.
[info] End nodes mapped from original model: '/bbox_head/conv_cls/Conv', '/bbox_head/conv_reg/Conv', '/bbox_head/conv_dir_cls/Conv'.
[info] Translation completed on ONNX model pointpillars_without_mid (completion time: 00:00:00.66)
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won't be optimized and compression won't be used) because there's less data than the recommended amount (1024), and there's no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] MatmulDecompose skipped
[info] Starting Mixed Precision
[info] Model Optimization Algorithm Mixed Precision is done (completion time is 00:00:00.06)
[info] LayerNorm Decomposition skipped
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████| 64/64 [02:25<00:00,  2.28s/entries]
[info] Model Optimization Algorithm Statistics Collector is done (completion time is 00:02:25.78)
[info] Starting Fix zp_comp Encoding
[info] Model Optimization Algorithm Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] Starting MatmulDecomposeFix
[info] Model Optimization Algorithm MatmulDecomposeFix is done (completion time is 00:00:00.00)
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.2158
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.1079
[info] No shifts available for layer pointpillars_without_mid/conv16/conv_op, using max shift instead. delta=0.4556
[info] No shifts available for layer pointpillars_without_mid/conv16/conv_op, using max shift instead. delta=0.2278
[info] No shifts available for layer pointpillars_without_mid/conv15/conv_op, using max shift instead. delta=0.1079
[info] Finetune encoding skipped
[info] Bias Correction skipped
[info] Adaround skipped
[info] Quantization-Aware Fine-Tuning skipped
[info] Layer Noise Analysis skipped
[info] Model Optimization is done
[info] Saved HAR to: /local/workspace/pointpillars_without_mid.har

Thanks,

2

64 seems to be a constant in here. I changed the model to resnet18 and resnet50 with a different input shape. I still get “Using dataset with 64 entries for calibration”.

3

Hi @ninfueng,

Calibration uses 64 images/data by default.
To use more data during calibration, please try using the model script to set the number of calibration images you want to use.
You can create a “.alls” file, e.g. model_script.alls and add the following line:

model_optimization_config(calibration, calibset_size=512)

Then, in your Python code, you load the model script and run the optimization.

model_script = "model_script.alls"
runner.load_model_script(model_script)
runner.optimize(calib_data)

You can also create the model script commands directly in your Python code. See the DataFlow Compiler User Guide for more details.

1 Like
4

Thank you @pierrem ,

This fixed the problem!