How can I compile yolov5m and my own yolov5 onnx to hef without quantization, optimization and nms

Another thing to check are the Anchors for the Yolov5 model.

Have a look at the alls script for the yolov5 model

GitHub Yolov5m alls script

It has the following line where it points to a JSON file:

nms_postprocess("../../../postprocess_config/yolov5m_nms_config.json", yolov5, engine=cpu)

This JSON file you can get from the Model Zoo. It is part of the “Pretrained” download. It uses the default anchors for the model. These can change during retraining.

GitHub - Hailo Model Zoo - Object detection - Hailo-8

You will need to extract the anchors from your model and adapt the JSON file. You can find the instruction here:

GitHub.com - Training Yolov5