Adaround is skipped when using optimization_level=4

General
1

Hi, I’m working on quantizing a neural network with optimization_level=4. However, Adaround (and other optimizations) are skipped without any error messages! The process works with optimization_level=2, but it doesn’t achieve the desired accuracy or SNR.

Has anyone encountered this issue? Could it be related to configuration, prerequisites for Adaround, or some tool-specific limitation?

1 Like
2

Hi @simon.eisenmann,

Welcome to the Hailo Community!
Could you please check the following?

  • GPU set up correctly: Make sure you have a recent GPU in your system in order to run Adaround. The Software Suite is compatible with many Nvidia GPUs, families please refer to the Installation section of the Hailo AI Software Suite User Guide. I suggest using the docker image, as it already includes the CUDA dependencies.
  • Number of images the calibration set: Adaround requires at least 1024 images to run. You can also trying forcing the algorithm execution adding the following command to the model script:
    post_quantization_optimization(adaround, policy=enabled)
3

Hi,

Thanks for the answer! I checked you suggestions. It seems that the GPU is set up correctly (I use the docker image) and the number of images in the calibration set is 1024! I also use the post_quantization_optimization(adaround, policy=enabled) command.

This is the only warning I get:

[Warning] CUDNN version should be 8.9 or higher, found 9.6.
Component Requirement Found
========== ========== ========== ==========
OS Ubuntu Ubuntu Required
Release 20.04 22.04 Required
Package python3-tk V Required
Package graphviz V Required
Package libgraphviz-dev V Required
Package python3.10-dev V Required
RAM(GB) 16 62 Required
RAM(GB) 32 62 Recommended
CPU-Arch x86_64 x86_64 Required
CPU-flag avx V Required
GPU-Driver 525 560 Recommended
CUDA 11.8 11.8 Recommended
CUDNN 8.9 9.6 Recommended
Var:CC unset unset Required
Var:CXX unset unset Required
Var:LD unset unset Required
Var:AS unset unset Required
Var:AR unset unset Required
Var:LN unset unset Required
Var:DUMP unset unset Required
Var:CPY unset unset Required

This warning makes no sense to me since the version 9.6 is higher than 8.9!

4

@simon.eisenmann The required CUDA and CUDNN versions are determined by the Tensorflow version used by the DFC (2.12 for recent releases).
Build from source | TensorFlow

  • Do you get any warnings when starting the optimization? The log should tell you what optimization_level is being used
  • Do you see any activity on the GPU during the optimization phase? You can open another terminal and check the GPU utilization with nvtop
  • I suggest running another test with the docker suite, which contains the exact CUDA and CUDNN versions required by Tensorflow. Since it is a docker, it will not break your existing environment, so it is worth trying
5

Thanks @pierrem for the tipps!

I tried your suggestions:

  • I do not get any warnings except this warning which I mentioned above: “[Warning] CUDNN version should be 8.9 or higher, found …”

  • I use optimization_level=4. The log tells me that the model script was loaded successfully. Here are some of the model script lines:

    1. model_optimization_flavor(optimization_level=4)
2. quantization_param({*}, precision_mode=a16_w16)
3. post_quantization_optimization(adaround, policy=enabled)

  • I see some high GPU activity during the calibration phase. However, the adaround optimization is skipped. I used nvidia-smi for monitoring the GPU utilization.

  • I currently use the docker suite. So the CUDA and CUDNN versions should be correct automatically. I always used the docker installation but I still have these problems.

I run the quantization in AWS. Can you maybe recommend a suitable EC2 instance which meets all the necessary requirements? I am not 100 percent sure, if I use the optimal machine and maybe there is some incompatibility.

Thanks a lot and hopefully you have an idea what could go wrong in my case!

6

Hi @simon.eisenmann,

Thanks for the details. Can you please send us a log of the optimization process when trying to run Adaround?
Please clarify which GPU you are using, and how many data are present in the calibration set.

7

Hi @pierrem,

Thanks a lot for your response!

Here is the log of the optimization process:

2025-01-09 09:24:41,659 - INFO - mo_script_parser.py:82 - Model received quantization params from the hn

2025-01-09 09:24:49,429 - INFO - algorithm_base.py:140 - MatmulDecompose skipped

2025-01-09 09:24:49,819 - INFO - algorithm_base.py:143 - Starting Mixed Precision

2025-01-09 09:24:50,771 - INFO - algorithm_base.py:159 - Model Optimization Algorithm Mixed Precision is done (completion time is 00:00:00.95)

2025-01-09 09:24:51,465 - INFO - algorithm_base.py:143 - Starting LayerNorm Decomposition

2025-01-09 09:25:00,645 - INFO - stats_collection.py:77 - Using dataset with 1024 entries for calibration

2025-01-09 09:27:20,288 - INFO - algorithm_base.py:159 - Model Optimization Algorithm LayerNorm Decomposition is done (completion time is 00:02:28.82)

2025-01-09 09:27:23,484 - INFO - algorithm_base.py:143 - Starting Statistics Collector

2025-01-09 09:27:23,486 - INFO - stats_collection.py:77 - Using dataset with 1024 entries for calibration

2025-01-09 09:36:30,473 - INFO - algorithm_base.py:159 - Model Optimization Algorithm Statistics Collector is done (completion time is 00:09:06.99)

2025-01-09 09:36:30,642 - INFO - stats_collection.py:77 - Using dataset with 1024 entries for calibration

2025-01-09 09:46:49,030 - INFO - algorithm_base.py:143 - Starting Fix zp_comp Encoding

2025-01-09 09:46:49,032 - INFO - algorithm_base.py:159 - Model Optimization Algorithm Fix zp_comp Encoding is done (completion time is 00:00:00.00)

2025-01-09 09:46:49,033 - INFO - algorithm_base.py:140 - Matmul Equalization skipped

2025-01-09 09:46:49,575 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv1/conv_op, using max shift instead. delta=0.4552

2025-01-09 09:46:49,659 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv1/conv_op, using max shift instead. delta=0.2276

2025-01-09 09:46:50,001 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc2/conv_op, using max shift instead. delta=0.6107

2025-01-09 09:46:52,507 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc2/conv_op, using max shift instead. delta=0.3054

2025-01-09 09:46:54,491 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv10/conv_op, using max shift instead. delta=1.1612

2025-01-09 09:46:54,619 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv10/conv_op, using max shift instead. delta=0.5806

2025-01-09 09:46:54,703 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv23/conv_op, using max shift instead. delta=2.1230

2025-01-09 09:46:54,795 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv22/conv_op, using max shift instead. delta=0.7985

2025-01-09 09:46:54,924 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv22/conv_op, using max shift instead. delta=0.3992

2025-01-09 09:46:55,262 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv10/conv_op, using max shift instead. delta=0.5806

2025-01-09 09:46:55,346 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv23/conv_op, using max shift instead. delta=1.0615

2025-01-09 09:46:55,752 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc11/conv_op, using max shift instead. delta=1.4782

2025-01-09 09:46:55,863 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc11/conv_op, using max shift instead. delta=0.7391

2025-01-09 09:46:55,983 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc12/conv_op, using max shift instead. delta=0.8179

2025-01-09 09:46:56,096 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc12/conv_op, using max shift instead. delta=0.4090

2025-01-09 09:46:56,168 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/fc11/conv_op, using max shift instead. delta=0.7391

2025-01-09 09:46:56,734 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv14/conv_op, using max shift instead. delta=0.1100

2025-01-09 09:46:56,858 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv14/conv_op, using max shift instead. delta=0.0550

2025-01-09 09:46:57,484 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv16/conv_op, using max shift instead. delta=0.6966

2025-01-09 09:46:57,610 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv16/conv_op, using max shift instead. delta=0.3483

2025-01-09 09:46:57,822 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv14/conv_op, using max shift instead. delta=0.0550

2025-01-09 09:46:59,197 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv22/conv_op, using max shift instead. delta=0.3992

2025-01-09 09:46:59,324 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv24/conv_op, using max shift instead. delta=0.8998

2025-01-09 09:46:59,421 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv24/conv_op, using max shift instead. delta=0.4499

2025-01-09 09:46:59,545 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv10/conv_op, using max shift instead. delta=0.5806

2025-01-09 09:46:59,630 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv23/conv_op, using max shift instead. delta=1.0615

2025-01-09 09:46:59,703 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv25/conv_op, using max shift instead. delta=0.6798

2025-01-09 09:46:59,794 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv25/conv_op, using max shift instead. delta=0.3399

2025-01-09 09:46:59,857 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv24/conv_op, using max shift instead. delta=0.4499

2025-01-09 09:47:00,044 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv28/conv_op, using max shift instead. delta=1.4063

2025-01-09 09:47:00,135 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv28/conv_op, using max shift instead. delta=0.7031

2025-01-09 09:47:00,259 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv29/conv_op, using max shift instead. delta=1.9203

2025-01-09 09:47:00,350 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv29/conv_op, using max shift instead. delta=0.9601

2025-01-09 09:47:01,714 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization1/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:01,808 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization1/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:02,524 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv30/conv_op, using max shift instead. delta=1.6578

2025-01-09 09:47:02,617 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv30/conv_op, using max shift instead. delta=0.8289

2025-01-09 09:47:03,561 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization2/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:03,655 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization2/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:04,376 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv31/conv_op, using max shift instead. delta=2.0050

2025-01-09 09:47:04,469 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv31/conv_op, using max shift instead. delta=1.0025

2025-01-09 09:47:05,410 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization3/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:05,505 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization3/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:06,212 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv32/conv_op, using max shift instead. delta=1.2180

2025-01-09 09:47:06,304 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv32/conv_op, using max shift instead. delta=0.6090

2025-01-09 09:47:07,254 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization4/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:07,353 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization4/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:08,086 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv33/conv_op, using max shift instead. delta=2.0603

2025-01-09 09:47:08,179 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv33/conv_op, using max shift instead. delta=1.0302

2025-01-09 09:47:09,123 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization5/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:09,217 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization5/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:09,948 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv34/conv_op, using max shift instead. delta=0.7450

2025-01-09 09:47:10,040 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv34/conv_op, using max shift instead. delta=0.3725

2025-01-09 09:47:11,508 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv35/conv_op, using max shift instead. delta=1.3662

2025-01-09 09:47:11,599 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv35/conv_op, using max shift instead. delta=0.6831

2025-01-09 09:47:13,075 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv36/conv_op, using max shift instead. delta=1.4131

2025-01-09 09:47:13,165 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv36/conv_op, using max shift instead. delta=0.7065

2025-01-09 09:47:13,748 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization8/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:13,844 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv_var_inv_layer_normalization8/conv_op, using max shift instead. delta=1.0000

2025-01-09 09:47:14,565 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv37/conv_op, using max shift instead. delta=0.5330

2025-01-09 09:47:14,657 - INFO - conv_stripped_op.py:614 - No shifts available for layer driveblocks_obdet/conv37/conv_op, using max shift instead. delta=0.2665

2025-01-09 09:49:08,578 - INFO - algorithm_base.py:140 - Finetune encoding skipped

2025-01-09 09:49:08,596 - INFO - algorithm_base.py:140 - Bias Correction skipped

2025-01-09 09:49:08,624 - INFO - algorithm_base.py:140 - Adaround skipped

2025-01-09 09:49:08,636 - INFO - algorithm_base.py:140 - Quantization-Aware Fine-Tuning skipped

We usually use calibset_size=1024, but we also tried 2500 for example.

Here are the details of the GPU (nvidia-smi):

Tue Jan 28 02:45:06 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:1E.0 Off |                    0 |
| N/A   26C    P8               9W /  70W |      0MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+