Is there a way to batch the calibration images during quantization?

General
1

The optimization/quantization process requires at least 1024 calibration images. The Jupyer Notebook tutorials show how to save the calibration images into a .npy file. The issue is that I cannot load all 1024 images into RAM. I have 16 GB of RAM. I am aware that this is below the recommended value, but I think it should be possible to batch the images so that they don’t have to be all loaded into RAM at the same time.

I was looking at the docs here:

and it mentions several options. I tried to get the tensorflow dataset option running, but the docs state that “image_info can be an empty dict for the quantization”. I don’t think Tensors can have Tensors and Dictionaries mixed together like that.

Looks like the .npy approach is better supported.
Do all calibration images need to be in a single .npy file? I noticed that there is a npy_dir option. Could I save each batch as an individual .npy file? Does the optimizer load the entire .npy file into RAM to use it?

1 Like
2

Hey @uriel.gonzalez ,

Yes, you can use the npy_dir option in DFC to batch the calibration images and avoid loading them all into RAM at once. Here’s how you can do it:

  1. Split your calibration dataset into smaller batches and save each batch as a separate .npy file. For example, if you have 1024 images, you can create batches of 100 images per .npy file.

  2. Use a script like this to split your dataset into multiple .npy files:

import numpy as np
import os

calibration_images = "path_to_your_calibration_images"
output_dir = "path_to_output_npy_dir"
batch_size = 100

os.makedirs(output_dir, exist_ok=True)

image_paths = sorted(os.listdir(calibration_images))
batches = [image_paths[i:i+batch_size] for i in range(0, len(image_paths), batch_size)]

for i, batch in enumerate(batches):
    images = [load_image(os.path.join(calibration_images, image_path)) for image_path in batch]
    np.save(os.path.join(output_dir, f"batch_{i}.npy"), np.array(images))
  1. Configure the DFC to use the npy_dir option and point it to the directory containing your batched .npy files:
hailomz compile my_model \
    --ckpt=model.onnx \
    --hw-arch=hailo8l \
    --calib-path=/path_to_output_npy_dir \
    --calib-format=npy_dir \
    --classes=1

By using the npy_dir option, the Hailo Dataflow Compiler will process the .npy files sequentially, avoiding the need to load all calibration images into memory at once. This approach is well-suited for machines with limited RAM.

The TensorFlow dataset option, while supported, may be more complex to set up correctly. If you prefer to use it, ensure that your dataset returns individual tensors rather than mixed types like dictionaries and tensors.