Hailo Model Zoo YOLOv8 retraining environment

General
,
1

Hi, I trained yolov8 with custom dataset on local python environment(as conda…), not docker and I got 0.97 mAP50 score from my test dataset with my custom model (custom_yolov8n.onnx).

Then I compiled onnx to hef using hailo model zoo yolov8n config(input size and classes are modified) and got 0.55 mAP50 score with same test dataset in hailo8.

So, my questions are

  1. Is this score typical?
  2. If I retrain custom model in docker environment provided in hailo model zoo, then can I got higher score?

I want to know if there is a difference between local python env(with ultralytics) and hailo model zoo docker env for retraining

2

Welcome to the Hailo Community @mmmsk!

There seems to be a significant accuracy decrease. This means there’s noise resulting from the quantization.

It is unlikely that this would have an effect.

Here are a few things you can try:

  1. Make sure you are using as a base the ModelZoo model script for the quantization
  2. Add this solution to your model script.
  3. Add more images to your calibset
  4. Try using Adaround as a post-quant alg
  5. Set the output to 16-bit - please note that this negatively affects the FPS
3

Thank you for the answer :slight_smile:
I tried compile with optimization_level=4 for Adaround and use 8000 calibset for calibration/finetuning.
I got 0.65 mAP50 score with this methods.

But I’m still wondering if there’s a way to improve my score.

4

Hi, how did you test the accuracy of the HEF file ?
Are you applying any normalization to the input ? Normally you shouldn’t because the model zoo script automatically adds a normalization layer to the model. The input should be uint8 [0…255].

5

Of course, I understand that there is a normalization layer command in the .alls file. So I use uint8 images for hef model input.
Now, I’m trying compile without normalization layer in .alls file.

6

Could you show us your full final model script with all of the added recommended commands?

Actually, @victorc meant that the normalization shouldn’t be performed outside of the compiled model. The recommended approach is to leave the normalization command in the model script.

7

@nina-vilela
This is my scripts for compile custom retrained yolov8n model
(Input size=1280x1280, Classes=6, DFCversion=3.27.0)

# custom_yolov8n.yaml
base:
- base/yolov8.yaml
postprocessing:
  device_pre_post_layers:
    nms: true
  hpp: true
network:
  network_name: yolov8n
paths: 
  alls_script: custom_yolov8n.alls
parser:
  nodes:
  - null
  - - /model.22/cv2.0/cv2.0.2/Conv
    - /model.22/cv3.0/cv3.0.2/Conv
    - /model.22/cv2.1/cv2.1.2/Conv
    - /model.22/cv3.1/cv3.1.2/Conv
    - /model.22/cv2.2/cv2.2.2/Conv
    - /model.22/cv3.2/cv3.2.2/Conv
quantization:
  calib_set:
  - models_files/my_dataset/adm_bbox_v2_calib_8000.tfrecord
preprocessing:
  network_type: detection
  input_shape:
  - 1280
  - 1280
  - 3
  meta_arch: yolo_v5
  padding_color: 114

.

# custom_yolov8n.alls
normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])

model_optimization_flavor(optimization_level=4)

change_output_activation(conv42, sigmoid)
change_output_activation(conv53, sigmoid)
change_output_activation(conv63, sigmoid)

performance_param(compiler_optimization_level=max)

nms_postprocess("../../postprocess_config/custom_yolov8n_nms_config_1280.json", meta_arch=yolov8, engine=cpu)

model_optimization_config(calibration, batch_size=16, calibset_size=8000)
post_quantization_optimization(finetune, policy=enabled, dataset_size=8000, epochs=8, learning_rate=0.0001)

.

# custom_yolov8n_nms_config_1280.json
{
	"nms_scores_th": 0.2,
	"nms_iou_th": 0.7,
	"image_dims": [
		1280,
		1280
	],
	"max_proposals_per_class": 100,
	"classes": 6,
	"regression_length": 16,
	"background_removal": false,
	"bbox_decoders": [
		{
			"name": "yolov8n/bbox_decoder41",
			"stride": 8,
			"reg_layer": "yolov8n/conv41",
			"cls_layer": "yolov8n/conv42"
		},
		{
			"name": "yolov8n/bbox_decoder52",
			"stride": 16,
			"reg_layer": "yolov8n/conv52",
			"cls_layer": "yolov8n/conv53"
		},
		{
			"name": "yolov8n/bbox_decoder62",
			"stride": 32,
			"reg_layer": "yolov8n/conv62",
			"cls_layer": "yolov8n/conv63"
		}
	]
}

So by compiling the above script, I was able to get a maximum mAP50 score of 0.65.
Now, I’m trying to optimize with full-16-bit and add this solutions as you recommanded.

8

@nina-vilela
Does the difference in onnx opset version affect the performance?

In the retraining example in docker of Hailo Model zoo, Onnx opset=11 is set, but in my case, onnx opset=17 is set.
As far as I know, my DFC version 3.27 supports onnx opset 17.
Is there a problem here?

9

Make sure to add to your model script the command below:

quantization_param([conv42, conv53, conv63], force_range_out=[0.0, 1.0])

As recommended in this solution

Short answer is no