Issue with YOLOv11 Quantization-Aware Fine-Tuning: total_distill_loss: nan

I am currently trying to compile the official YOLOv11 model using my own dataset for calibration, but I am encountering an issue.

Setup:

  • YOLO version: yolov11l

  • Model Zoo version: 2.15

  • Number of calibration images: 310

During the calibration, the logs show the following repeatedly:
oss_yolov11l/conv159: nan - _distill_loss_yolov11l/conv117: nan - … - total_distill_loss: nan

After the Layer Noise Analysis step, the process fails with the following traceback:


Traceback (most recent call last):

raise BackendValueError(f"Unsupported NaN values were found in params {not_good_params}")

Command used:

hailomz compile yolov11l --ckpt ./yolo11l.onnx --hw-arch hailo8 --calib-path ./seoul640/ --classes 80

Everything seems to run normally until the total_distill_loss becomes nan, which then causes the error.

I have verified that my calibration dataset is fine and the ONNX file also appears correct. Using the same version of the Model Zoo, I have successfully converted YOLOv7 and YOLOv8 models without issues. However, with YOLOv11, this problem occurs consistently.

Has anyone else encountered this issue or have any suggestions for resolving it?