Cannot detect YOLOX with quantization model trained on custom dataset

General
, ,
1

We trained on a custom dataset using hailo-ai’s YOLOX repository.
With that model, the YOLOX objectness confidence output is less than 0.03, making it undetectable.

With the pre-trained onnx model described in the yaml file, we are evaluating in an environment where the YOLOX objectness confidence output is 0.8 or higher.

I’m assuming there is no problem with the conversion command or post-processing, since it works when hef is created from the pre-trained model onnx.

Comparing the DFC logs for the pre-training model and the custom training model, there are differences in the following logs, which may indicate that the output ranges may not be adjusted properly.

Custom training models tend to have a lower range of output.

  • [pretrain model dfc log]
[info] Statistics Collector is done (completion time is 00:00:25.00)
[info] Output layer yolox_tiny/conv54 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [4.0418141135885387e-16, 0.9451166987419128]).
[info] Output layer yolox_tiny/conv56 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [6.123049636213117e-17, 0.9960691332817078]).
[info] Output layer yolox_tiny/conv68 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [6.643642703040511e-17, 0.9771779775619507]).
[info] Output layer yolox_tiny/conv70 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [7.638780287564512e-14, 0.9980177283287048]).
[info] Output layer yolox_tiny/conv81 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [1.4554710518407954e-12, 0.9749641418457031]).
[info] Output layer yolox_tiny/conv83 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [5.122564772364058e-08, 0.9991075992584229]).
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] Finetune encoding skipped
  • [custom model dfc log]

[info] Statistics Collector is done (completion time is 00:00:24.69)
[info] Output layer yolox_tiny/conv54 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [1.831859322010132e-07, 0.6948390603065491]).
[info] Output layer yolox_tiny/conv56 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [0.0015948283253237605, 0.2364097535610199]).
[info] Output layer yolox_tiny/conv68 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [2.5630305169888068e-12, 0.7390304207801819]).
[info] Output layer yolox_tiny/conv70 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [0.0036237630993127823, 0.3684675693511963]).
[info] Output layer yolox_tiny/conv81 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [1.015656447833635e-08, 0.9411818981170654]).
[info] Output layer yolox_tiny/conv83 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [0.012703766115009785, 0.424691766500473]).
[info] Starting Fix zp_comp Encoding
[info] Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] No shifts available for layer yolox_tiny/conv1/conv_op, using max shift instead. delta=0.7187
[info] No shifts available for layer yolox_tiny/conv1/conv_op, using max shift instead. delta=0.3593
[info] Finetune encoding skipped

Would appreciate your advice on DFC options and other points to correct or check.

2

Hey @shinji.uchinoura ,

Welcome to the Hailo Community!

To resolve the issue of very low objectness confidence outputs from your custom YOLOX model :

  1. Verify your calibration dataset matches your training data distribution. This ensures the Dataflow Compiler (DFC) adjusts ranges properly. See the DFC Calibration Tutorial for details.
  2. Analyze quantization using hailo profiler to check your custom model’s weight and activation distributions:
hailo profiler --input custom_model.onnx --output profiler_output.json

Compare these to the pre-trained model to spot differences.

  1. Use DFC’s range enforcement options to fine-tune output behavior, e.g., bounding sigmoid activation layers to [0, 1]. Check the DFC user guide’s Quantization and Range Adjustment section for more info.