Layer Noise Analysis: layout failed: INVALID_ARGUMENT

General Discussion
,
1

Hello!

I’ve faced some issues with Layer Noise Analysis while trying to optimize_full_precision for yolov8:

[info] Starting Layer Noise Analysis`
Full Quant Analysis: 0%| | 0/16 [00:00<?, ?iterations/s]E0000 00:00:1764262369.223821 2508 meta_optimizer.cc:966] layout failed: INVALID_ARGUMENT: Size of values 1 does not match size of permutation 4 @ fanin shape inlat_model_1/normalization1_5/act_op_1/SelectV2-1-TransposeNHWCToNCHW-LayoutOptimizer

At the same time, Quantization-Aware Fine-Tuning is done correctly (using GPU)

My environment:

  • Ubuntu 22.04, Kernel: 6.8.0-87-generic
  • NVIDIA TITAN X (Pascal
  • nvidia-driver-535
  • cuda-12.5
  • cudnn-9.16
  • tensorflow[and-cuda] 2.18.0
  • DFC 3.33.0

I am using yolov8s from hailo_model_zoo and the following code:

from hailo_sdk_client import ClientRunner

def main(har_model, calib_dataset):
  runner = ClientRunner(har=har_model)
  
  ### yolov8s
  alls = """
normalization1 = normalization(\[0.0, 0.0, 0.0\], \[255.0, 255.0, 255.0\])
model_optimization_config(calibration, batch_size=1)
change_output_activation(conv42, sigmoid)
change_output_activation(conv53, sigmoid)
change_output_activation(conv63, sigmoid)
nms_postprocess(‘./nms_layer_config_y8s.json’, meta_arch=yolov8, engine=cpu)
post_quantization_optimization(finetune, policy=enabled, learning_rate=0.000025)
performance_param(compiler_optimization_level=1)
“”"
  
  runner.load_model_script(alls)
  runner.optimize_full_precision(calib_data=calib_dataset)
  runner.optimize(calib_dataset)

  runner.save_har(quantized_har_model)
2

Hey @Osborn_Odonovon,

Welcome to the Hailo Community!

The error originates from TensorFlow’s layout optimizer during Layer Noise Analysis on GPU, not from Hailo itself. This is a known issue when TensorFlow encounters transpose permutation problems.

Quick fixes:

  1. Disable Layer Noise Analysis – It’s optional and mainly for debugging. Add this to your model script:

    model_optimization_config(checker_cfg, policy=disabled)

    Your optimize() and QFT will still run normally.
    Or at least reduce its load:

    model_optimization_config(checker_cfg, batch_size=1)

    This is explicitly suggested when Layer Noise Analysis on GPU runs into memory / layout issues.

  2. Force CPU instead of GPU – Run optimization without GPU access:

    CUDA_VISIBLE_DEVICES="" python your_script.py

    This avoids the TensorFlow layout issue while keeping your QFT results.

  3. Proceed without analysis – Since your QFT succeeded, you can skip Layer Noise Analysis entirely, save your quantized HAR, and compile to HEF for hardware evaluation.

The safest path is option 1 or 2. Let us know if you hit any other issues!