Hi all, let’s talk quantization.
Most quantization acceptance checks I’ve seen compare the int8 build to the float model on threshold-free metrics: accuracy, top-1, mAP, precision@k. But a deployed model almost never runs threshold-free. A detector has a score_threshold in its postprocess config, a verifier has a similarity cutoff, a classifier has a confidence below which a human takes over. That constant was fitted to the float model, and nothing in the usual gate re-checks it after quantization.
So my question for people shipping int8 on Hailo: how do you decide a build is good enough?
Do you re-derive the threshold on the quantized model, or keep the one from fp32?
Does anything in your acceptance check depend on where the threshold sits (false accepts/rejects, false alarms per hour, precision/recall at the configured cutoff), or is it accuracy/mAP against the float model?
One case I measured, for context: an embedding model (ArcFace R50) through the Dataflow Compiler on RPi 5 + Hailo-8. The ranking metrics barely moved (rank-1 0.990 → 0.985). At the inherited fp32 threshold, false accepts went 30 → 104 of 10,000 held-out impostor pairs, and false rejects rose too. Keeping first/last layers in 16-bit didn’t change that. Re-deriving the threshold on the int8 build cut false accepts to 44, at the cost of more false rejects. One encoder and one small sample, so the size is specific to that build. Whether detectors or classifiers drift as much is exactly what I don’t know.
Write-up: https://towardsdatascience.com/the-number-your-quantization-gate-doesnt-have/
Code and released embeddings: GitHub - maxeffectus/quantization-threshold-drift: Companion code for held-out FAR/FRR after quantization: the threshold the acceptance gate does not read. · GitHub
I would be very interested to hear about your experience!
P.S. if you know a public guide or acceptance doc that gates on a threshold-dependent number, I’d like to add it to the survey in the article.