I compiled 2 different models of Hailo on 2 different datasets. One dataset has very low lightning conditions, other dataset has good lightning condition.
I had to use optimization level 4, with 1024 images to get the best performance. However, still I get some performance drop. I made sure that I cover all types of defects to create 1024 calibration dataset.
1 model gives me 1.5% drop compared to the original pytorch model accuracy, and the other model with low lightning condition gives me 8% drop on Hailo.
Why am I getting results like this? With same optimization setup, ideally both the models should give similar performance. Adaround optimization with greater than 1024 images, takes huge amount of time, and is ideally not feasible.
Both the models are exactly same custom models. Also, the image resolution of both the dataset is same.
So you’re seeing an accuracy drop with low-light images - this is pretty common actually. Here’s what’s likely happening:
Lighting really messes with how features look to the model. Low-light images bring in a lot of noise, kill the contrast, and can shift colors around. All of this makes it harder for the model to pick up on the important details it was trained to recognize.
The quantization process gets tricky with low-light data. Even though you’re using Adaround with 1024 calibration samples, low-light scenes compress the pixel values into a smaller range. The subtle features that matter for defect detection become even more subtle, and quantization has a harder time preserving them accurately.
Same model, different behavior after quantization. Your architecture might be identical, but if one dataset has significantly different characteristics (like yours with low-light conditions), the quantization can hit it differently. That 8% drop suggests your low-light model relies heavily on those fine details that get lost in the quantization process.
I’d recommend mixing your good lighting and low-light images into the same dataset for calibration. That way, the quantization process can learn how to handle both lighting conditions properly and you should see better performance across the board!