`KeyError: 'stats/preact/dynamic_histogram'` during quantization

General
, ,
1

During optimization of almost any model I face the following error right after calibration:

File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py", line 1674, in optimize
    self._optimize(calib_data, data_type=data_type, work_dir=work_dir)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py", line 1542, in _optimize
    self._sdk_backend.full_quantization(calib_data, data_type=data_type, work_dir=work_dir)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 910, in full_quantization
    self._full_acceleras_run(self.calibration_data, data_type)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1034, in _full_acceleras_run
    optimization_flow.run()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 246, in run
    self.step1()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py", line 63, in parent_wrapper
    func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 256, in step1
    self.core_quantization()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 298, in core_quantization
    self._create_hw_params()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/flows/optimization_flow.py", line 335, in _create_hw_params
    create_hw_params.run()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 47, in run
    return super().run()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 120, in run
    self._run_int()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 318, in _run_int
    comp_to_retry = self._create_hw_params_component(matching_component_group)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 197, in _create_hw_params_component
    retry_negative_exp_list = self._hanlde_negative_exponent(layer, matching_component_group)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/create_hw_params/create_hw_params.py", line 209, in _hanlde_negative_exponent
    algo.run()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/optimization_algorithm.py", line 47, in run
    return super().run()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/algorithm_base.py", line 120, in run
    self._run_int()
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 70, in _run_int
    l_fix = self.fix_output(l_fix)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 112, in fix_output
    l_fix = self._split_layer_solution(l_fix)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/neg_exp_fixer.py", line 137, in _split_layer_solution
    a_layer, b_layer = splitter.split_layer(layer, auto_clip=self._check_feature(cfg.auto_clip), rank=cfg.rank)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/layer_splitter.py", line 65, in split_layer
    a_stats, b_stats = self._propagate_stats(original_stats, auto_clip=clip_range)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/layer_splitter.py", line 115, in _propagate_stats
    a_stats = create_stats(original_stats, map_a)
  File "/local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_model_optimization/algorithms/neg_exponent_fixer/layer_splitter.py", line 112, in create_stats
    res[key] = stats[new_key]
KeyError: 'stats/preact/dynamic_histogram'

I’m using hailo-dataflow-compiler==3.27.0in a hailo_ai_sw_suite_2024-04 container. The models I’ve tried to quantize:
resnext50 - success
resnext101 - KeyError
efficientnetv2s - KeyError
regnet_y_1.6f - KeyError

I was thinking the problem occurs because of global_avgpool_reduction or because converting of global average pooling layers to fp16, but then I faced the same error for resnext101 which didn’t include any of those in the optimization script. Could somebody please help with this?

2

Hi @Kirill_Rybachuk,
Can you please share a bit more info?

  1. Are you starting from a 32b float model?
  2. What is the exact command that you are using for the conversion?
    Global Average Pools are not so friednly, if they occur in many places through the model. You can check the “tiled Squeeze and Excitation” operator (there is also a paper from our ML folks explaning all about it:) ).

Basically we show that instead of taking the whole tensor into the GAP, you can take only some lines from the image, and get very close result while boosting your runtime. Check out the chapter called “se_optimization” in our Dataflow Compiler Userguide.

3
  1. yes
  2. afaik, there are no SE blocks in resnext, and I didn’t include any processing for avgpools in the optimization script, that’s why I think the error is not related to avgpools. On top of that, when I did have problems with global avg pools in the past, the error looked different (usually there were problems with shift delta that could be solved with global_avgpool_reduction or se_optimization like you said), which led me to thought there is a different case for this error.

My optimization script for resnext101:

        model_script_lines = [
            "normalization1 = normalization([123.675, 116.28, 103.53], [58.395, 57.12, 57.375])\n"
            "quantization_param(output_layer1, precision_mode=a16_w16)\n",
            "pre_quantization_optimization(weights_clipping, layers={conv*}, mode=percentile, clipping_values=[0.1, 99.9])\n",  
            "pre_quantization_optimization(activation_clipping, layers={conv*}, mode=percentile, clipping_values=[0.0, 99.9])\n", 
            f"model_optimization_config(calibration, calibset_size={calib_size})\n",
            f"model_optimization_flavor(optimization_level={optimization_level}, compression_level=0, batch_size=1)\n",
4

Thanks, for the info, one more question, what is the size of the calibset?

5

I tried 1024 for optimization_level=2 and a range of sizes from 128 to 1024 for optimization_level=1

6

OK,
For the calibration we highly recomment not to surpass the 64 images. This is the for the basic statistics collection. I believe that if you would keep the default value of 64, you would also not need to clip the values.

I ran now a quick text on resnext101 and regnet_y_1.6g and was able to optimize (quantize) both. For the Efficientnet arch, I would like to suggest to take a look at the variants that we’ve already converted in the model-zoo.

I’ve used this normalization in the optimization process:
normalization([123.675, 116.28, 103.53], [58.395, 57.12, 57.375])

7

64 images only work for optimization_level<=1. AdaRound wants at least 1024. Quantization with 64 without adaround did succeed but accuracy dropped by ~10% which is too much. So I think we need a customized fine-tuned checkpoint converted with AdaRound

8

I’m sorry for not being clear. for the calibration part, keep the default of 64 images. For the rest of the algorithms, definitely use more.

9

How do I specify different sample sizes for calibration and AdaRound in the alls config? There seems to be no parameter indicating sample size in model_optimization_flavor and I can’t find any documentation about all possible parameters there

10

or in post_quantization_optimization… in either case, I only see some tutorials or examples in the model zoo, but is there reference API where I can find all the parameters these functions can possibly accept?

11

If you use an npy archive, it’s size set the size for the advanced algorithm. If you’re using a generator, you can specifiy for AdaRound, like this:
post_quantization_optimization(adaround, policy=enabled, dataset_size=256)

1 Like
12

Unfortunately this also leads to KeyError: 'stats/preact/dynamic_histogram'. So every time I increase sample size for either calibration or adaround, I get this. Maybe we can get to the bottom of this particular error? It doesn’t look related to quantization process itself