Thanks for clarifying.
There are some known issues with the quantization of yolov8 re-trained with few classes. You can try adding this solution to your model script.
Besides that, you can also try using some post-quant algorithms such as adaround. This can also be enabled with a command on the model script, for example:
post_quantization_optimization(adaround, policy=enabled)
You can read more about adaround and other post-quant algs here.