How can I compile yolov5m and my own yolov5 onnx to hef without quantization, optimization and nms

General
,
1

I can compile hailo yolov5m.onnnx and my own i3yolov5m onnx to hef with nms layer, and get the detection results. But when I integrate the hailo detection to my test application, and run over the test cases, the accuracy drops obviously comparing with tensorRT model which is running on GPU. (the tensorRT engine is built from same i3yolov5m.onnx)

Following the tutorial, I’m trying to build the onnx to hef without quantization and optimization. But seems I cannot. I I use the following code to build.

nnx_model_name = ‘i3yolov5m23’
onnx_path = ‘…/models/i3yolov5m23.onnx’
assert os.path.isfile(onnx_path), ‘Please provide valid path for ONNX file’

Initialize a new client runner

runner = ClientRunner(hw_arch=‘hailo8’)

Any other hw_arch can be used as well.

Translate YOLO model from ONNX

runner.translate_onnx_model(onnx_path, onnx_model_name,
start_node_names=[‘images’],
end_node_names=[‘Conv_344’, ‘Conv_309’, ‘Conv_274’],
net_input_shapes={‘images’: [1, 3, 640, 640]})

calib_dataset = np.load(‘i3calib_set_640_640.npy’)

Call Optimize to perform the optimization process

runner.optimize(calib_dataset)

Save the result state to a Quantized HAR file

quantized_model_har_path = f’{onnx_model_name}_quantized_model_nms.har’
runner.save_har(quantized_model_har_path)

hef = runner.compile()

file_name = f’{model_name}.hef’
with open(file_name, ‘wb’) as f:
f.write(hef)

I found I have to invoke “runner.optimize(calib_dataset)”, otherwise when I compile it to hef, I get warning that there is no weights.

The hef I got could run with hailo8, and I get 3 output streams. But when I tested, the output box is messy. I tried to compile yolov5m from hailo website, and get same wrong boxes. And I found the qp_zp of the output steam is about 181, 182 or 184 of yolov5s. I downloaded the yolo_vehicle from hailo website, and it works correctly.

Can you please let me know how to compile yolov5m and my yolo from onnx to hef without quantization and optimization? And can you please uplaod me a yolov5m onnx without nms layer to test?

Thanks,
Dan

2

Quantizing/Optimizing a network is an essential part of running a model on the Hailo-8 AI accelerator. Neural networks are developed and trained using floating point because it keeps the process simple. No need to worry about data ranges. However floating point hardware uses larger silicon area and power. One point that makes the Hailo-8 so power efficient is computing the values using integer hardware. Hailo-8 supports 8-bit (default) and 4 and 16-bit.

Nothing in live is free, so there will be a accuracy degradation when quantizing a network. The goal is to be in the single digit percentage range and in many cases below 1%.

This may require to use more advanced optimization algorithms. For these the Hailo Dataflow Compiler will need a GPU.

You can run your model after parsing in the emulator at full precision to confirm the model was parsed correctly and the model is still working as expected.

3

Thanks for you reply Klausk!

I do have a GPU on my computer to run tensorRT engine below. I could compile my own yolom5 to hef and run on hailo8 well, but when I compare the accuracy with tensorRT engine which is compiled from same yolov5m, the accuracy drops. So I will go on to test with more advance and optimization algorithms. I do have a GPU GTX 1050Ti on my computer, and can I run it in WSL?

4

I’m not recommend compile yolov5m and your yolo from onnx to hef without quantization and optimization.

I’m using yolov5 without NMS Layers. In my experience, the NMS layer supported by Hailo was the cause of the processing speed and performance degradation.

using NMS source code from yolov5 github, or coding your custom NMS for coustom yolov5 model.

5

That’s very important suggestion. Thank you!

I tried to compile my onnx to hailo8 hef without nms, but I cannot get correct results from the output. The boxes I extracted from 3 output streams are not correct. Do you have any clue for it?

Many thanks!

6

I would recommend to first check the normalization. It should be added to the model in the model script and then you should not normalize the input data, otherwise you will apply it twice.

How do you check the result? On the hardware or the emulator?

7

I checked the result on hardware.

I tried a few models on hailo website. After some modification, of the yolo sample code, I can get correct boxes from yolov5_vehicles.hef which has no nms. But when I compile my own yolo or hailo yolomv5m, they output some boxes, but they are not correct.

Do you try the yolov5s.hef from hailo Website? If you can get correct results, I may write wrong code to extract boxes from the output streams.

8

Another thing to check are the Anchors for the Yolov5 model.

Have a look at the alls script for the yolov5 model

GitHub Yolov5m alls script

It has the following line where it points to a JSON file:

nms_postprocess("../../../postprocess_config/yolov5m_nms_config.json", yolov5, engine=cpu)

This JSON file you can get from the Model Zoo. It is part of the “Pretrained” download. It uses the default anchors for the model. These can change during retraining.

GitHub - Hailo Model Zoo - Object detection - Hailo-8

You will need to extract the anchors from your model and adapt the JSON file. You can find the instruction here:

GitHub.com - Training Yolov5

9

I found both yolov5m.hef and yolov5s.hef I downloaded from the URL are compiled with nms already. But the yolov5m_vehicles.hef is compiled without nms, so its output has 3 feature maps.

I double checked the anchors and they are all correct, but I still get wrong boxes.

Here is the yolov5s_no_nms.hef I compiled from yolov5s.onnx from hailo website.

hailo

Can you please help me to check the yolov5s_no_nms.hef, do I compile the hef wrongly?

Or, can you please send me a compiled yolov5m.hef or yolov5s.hef without nms, and output 3 feature maps?

Thanks a lot!

10

hi @KlausK - I’m having issues with poor performance with yolo5s on a custom dataset. The anchors were updated during training as it is a small object dataset.

I’m not quite sure where these anchors need to be added. Which yaml file do you refer to specifically, I only see anchors mentioned in cfg/base/yolo.yaml

I ran the script in the read me you linked to inspect the model and get these values: 
tensor([[[[[[ 2.27344,  4.06250]]],
          [[[ 2.77148,  6.73828]]],
          [[[ 4.26172,  7.46484]]]]],
        [[[[[ 4.53906, 11.07031]]],
          [[[ 7.21875,  9.75000]]],
          [[[ 6.37109, 15.98438]]]]],
        [[[[[ 9.89844, 14.13281]]],
          [[[11.42969, 23.01562]]],
          [[[19.93750, 29.39062]]]]]], dtype=torch.float16)

Any help would be greatly appreciated because right now with yolov5s, I have only nonsensical boxes of various sizes plastering the image, not a single legit detection. Given how different my anchors are to the defaults, this must be a big factor why.

Thanks!
NSN

11

Hi @natsayin_nahin
If you can provide the pytorch checkpoint, we can try to compile using our flow and see if the problem is solved.

12

hi @shashi thanks again for the help - I’ve added yolov5s_2class_visdrone.pt to the Drive folder here. Anything else please let me know.

13

Thanks @natsayin_nahin We will keep you posted on what we find. This should help a bunch of other folks as well.

1 Like
14

Forgot to update here - it was somewhat easy in the end but took a little guess work as to which file to update, especially given the confusing info message when parsing:

[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.34)
[info] NMS structure of yolov5 (or equivalent architecture) was detected. Default values of NMS anchors were loaded to NMS config json

that lead me down the road of updating the base/yolo.yaml file - however that wasn’t required at all.

The only file needing updated was the nms_config. I was looking for some key:value called anchor, but in the nms config these key:value pairs are in the "bbox_decoders" under "w" and "h"

The updated yolov5_nms_config.json now looks like this (bear in mind that I have a custom class number for my 2classes+background=3)

Updated file 
{
  "nms_scores_th": 0.001,
  "nms_iou_th": 0.6,
  "image_dims": [640, 640],
  "max_proposals_per_class": 80,
  "background_removal": false,
  "classes": 3,
  "bbox_decoders": [
      {
          "name": "bbox_decoder55",
          "w": [
              2.27344,
              2.77148,
              4.26172
          ],
          "h": [
              4.06250,
              6.73828,
              7.46484
          ],
          "stride": 8,
          "encoded_layer": "conv55"
      },
      {
          "name": "bbox_decoder63",
          "w": [
              4.53906,
              7.21875,
              6.37109
          ],
          "h": [
              11.07031,
              9.75000,
              15.98438
          ],
          "stride": 16,
          "encoded_layer": "conv63"
      },
      {
          "name": "bbox_decoder70",
          "w": [
              9.89844,
              11.42969,
              19.93750
          ],
          "h": [
              14.13281,
              23.01562,
              29.39062
          ],
          "stride": 32,
          "encoded_layer": "conv70"
      }
  ]
}

Unfortunately, the yolov5 also seems to have the same issue with nms and the overlapping boxes - but I’ll update regards to that in the other thread to keep this one on topic