Converting ONNX to HEF

General
1

I’ve been trying to follow this guide:

and I’m not having a lot of luck. I’m finding my way on this, but here’s what I’ve done:

I trained a custom model using yolov5s. When the model completed training, this summary was printed:

75 epochs completed in 14.914 hours.
Optimizer stripped from runs/detect/train/weights/last.pt, 18.5MB
Optimizer stripped from runs/detect/train/weights/best.pt, 18.5MB

Validating runs/detect/train/weights/best.pt...
Ultralytics 8.3.162 🚀 Python-3.9.22 torch-2.7.1+cu126 CUDA:0 (NVIDIA L4, 22574MiB)
YOLOv5s summary (fused): 84 layers, 9,112,310 parameters, 0 gradients, 23.8 GFLOPs
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95): 100%|██████████| 29/29 [00:17<00:00,  1.67it/s]
                   all       2854      11206      0.868      0.775      0.855      0.648
                person       2693      11004      0.868      0.673      0.804      0.577
                   cat        184        202      0.869      0.876      0.906      0.719
Speed: 0.1ms preprocess, 1.9ms inference, 0.0ms loss, 0.7ms postprocess per image

I’ve downloaded the docker container and have a shell running in it.

I put my converted onnx file into the shared_with_docker folder. I copied hailo_model_zoo/cfg/networks/yolov5s.yaml to shared_with_docker and added the # classes. The complete file looks like:

base:
- base/yolov5.yaml
network:
  network_name: yolov5s
evaluation:
  classes: 2
paths:
  alls_script: yolov5s.alls
  network_path:
  - models_files/ObjectDetection/Detection-COCO/yolo/yolov5s_spp/pretrained/2023-04-25/yolov5s.onnx
  url: https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/ObjectDetection/Detection-COCO/yolo/yolov5s_spp/pretrained/2023-04-25/yolov5s.zip
postprocessing:
  device_pre_post_layers:
    nms: true
  meta_arch: yolo_v5
  hpp: true
info:
  task: object detection
  input_shape: 640x640x3
  output_shape: 80x5x80
  operations: 17.44G
  parameters: 7.46M
  framework: pytorch
  training_data: coco train2017
  validation_data: coco val2017
  eval_metric: mAP
  full_precision_result: 35.33
  source: https://github.com/ultralytics/yolov5/releases/tag/v2.0
  license_url: https://github.com/ultralytics/yolov5/blob/master/LICENSE
  license_name: GPL-3.0

when I try to create the hef file, i get:

(hailo_virtualenv) hailo@reddwarf:/local/shared_with_docker$ hailomz compile --yaml yolov5s.yaml --ckpt person_cat.onnx --hw-arch hailo8l --calib-path calib --classes 2
<Hailo Model Zoo INFO> Start run for network yolov5s ...
<Hailo Model Zoo INFO> Initializing the hailo8l runner...
[info] Translation started on ONNX model yolov5s
[info] Restored ONNX model yolov5s (completion time: 00:00:00.08)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.29)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:00.64)
[info] According to recommendations, retrying parsing with end node names: ['/model.24/Concat_3'].
[info] Translation started on ONNX model yolov5s
[info] Restored ONNX model yolov5s (completion time: 00:00:00.06)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.27)
[info] NMS structure of yolov8 (or equivalent architecture) was detected.
[info] In order to use HailoRT post-processing capabilities, these end node names should be used: /model.24/cv2.0/cv2.0.2/Conv /model.24/cv3.0/cv3.0.2/Conv /model.24/cv2.1/cv2.1.2/Conv /model.24/cv3.1/cv3.1.2/Conv /model.24/cv2.2/cv2.2.2/Conv /model.24/cv3.2/cv3.2.2/Conv.
[info] Start nodes mapped from original model: 'images': 'yolov5s/input_layer1'.
[info] End nodes mapped from original model: '/model.24/Concat_3'.
[info] Translation completed on ONNX model yolov5s (completion time: 00:00:00.57)
[info] Translation started on ONNX model yolov5s
[info] Restored ONNX model yolov5s (completion time: 00:00:00.06)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.27)
[info] NMS structure of yolov8 (or equivalent architecture) was detected.
[info] In order to use HailoRT post-processing capabilities, these end node names should be used: /model.24/cv2.0/cv2.0.2/Conv /model.24/cv3.0/cv3.0.2/Conv /model.24/cv2.1/cv2.1.2/Conv /model.24/cv3.1/cv3.1.2/Conv /model.24/cv2.2/cv2.2.2/Conv /model.24/cv3.2/cv3.2.2/Conv.
[info] Start nodes mapped from original model: 'images': 'yolov5s/input_layer1'.
[info] End nodes mapped from original model: '/model.24/cv2.0/cv2.0.2/Conv', '/model.24/cv3.0/cv3.0.2/Conv', '/model.24/cv2.1/cv2.1.2/Conv', '/model.24/cv3.1/cv3.1.2/Conv', '/model.24/cv2.2/cv2.2.2/Conv', '/model.24/cv3.2/cv3.2.2/Conv'.
[info] Translation completed on ONNX model yolov5s (completion time: 00:00:00.57)
[info] Appending model script commands to yolov5s from string
[info] Added nms postprocess command to model script.
[info] Saved HAR to: /local/shared_with_docker/yolov5s.har
<Hailo Model Zoo INFO> Preparing calibration data...
[info] Loading model script commands to yolov5s from /local/workspace/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov5s.alls
[info] Loading model script commands to yolov5s from string
Traceback (most recent call last):
  File "/local/workspace/hailo_virtualenv/bin/hailomz", line 33, in <module>
    sys.exit(load_entry_point('hailo-model-zoo', 'console_scripts', 'hailomz')())
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main.py", line 122, in main
    run(args)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main.py", line 111, in run
    return handlers[args.command](args)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main_driver.py", line 248, in compile
    _ensure_optimized(runner, logger, args, network_info)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main_driver.py", line 91, in _ensure_optimized
    optimize_model(
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/core/main_utils.py", line 351, in optimize_model
    optimize_full_precision_model(runner, calib_feed_callback, logger, model_script, resize, input_conversion, classes)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/core/main_utils.py", line 337, in optimize_full_precision_model
    runner.optimize_full_precision(calib_data=calib_feed_callback)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2090, in optimize_full_precision
    self._optimize_full_precision(data_continer)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 2093, in _optimize_full_precision
    self._sdk_backend.optimize_full_precision(data_continer)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1722, in optimize_full_precision
    model, params = self._apply_model_modification_commands(model, params, update_model_and_params)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py", line 1610, in _apply_model_modification_commands
    model, params = command.apply(model, params, hw_consts=self.hw_arch.consts)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/script_parser/nms_postprocess_command.py", line 404, in apply
    pp_creator = create_nms_postprocess(
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/tools/core_postprocess/nms_postprocess.py", line 1824, in create_nms_postprocess
    pp_creator.prepare_hn_and_weights(
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/tools/core_postprocess/nms_postprocess.py", line 1641, in prepare_hn_and_weights
    super().prepare_hn_and_weights(hw_consts, engine, **kwargs)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/tools/core_postprocess/nms_postprocess.py", line 1119, in prepare_hn_and_weights
    self.add_postprocess_layer_to_hn()
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/tools/core_postprocess/nms_postprocess.py", line 1068, in add_postprocess_layer_to_hn
    raise NMSConfigPostprocessException(f"The layer {encoded_layer.name} doesn't have one output layer")
hailo_sdk_client.tools.core_postprocess.nms_postprocess.NMSConfigPostprocessException: The layer yolov5s/conv55 doesn't have one output layer

Can anyone give me a hand with this? It seems like I’m just inches from the finish line and I’m stuck!

Thanks!

2

Substitute this line in the model script:

with:
nms_postprocess(meta_arch=yolov5, engine=cpu)

With the change, the config file used for the nms will be the one identified by the parser for your model, with your amount of classes, and not the one available in the ModelZoo

3

Thanks, I tried that and it now gives:

hailo_sdk_client.sdk_backend.sdk_backend_exceptions.AllocatorScriptParserException: 
Cannot infer bbox conv layers automatically. 
Please specify the bbox layer in the json configuration file

I actually tried this yesterday, but I couldn’t edit my post while it was pending.

You know, there are two massive problems with this toolchain.

First, you never really explain to people that the tool chain is fragile, and you have to use the complete tool chain. You have lots of instructions that show using the complete toolchain, but don’t explain this. So, I look at it, and say. “Well now, I can do this part that way, and the other part their way”. Then it doesn’t work.

That’s the problem I’m hitting now. My onnx file was created using ultralytics and it’s not compatible.

The other problem is all of the model zoo docs show creating your own docker image, which is painful, but the rest of your universe is centered around your own docker image. Nobody wants to go through the pain creating a custom image and using that if it can be avoided. You just don’t say it can’t be avoided. The other thing is that the model zoo has inconsistent tools compared to the complete docker image.

The concepts of the model zoo and what it’s really doing is never explained. Perhaps real ML people know this stuff, but you could really hire a technical writer and work on your docs to make the tools more accessible. Just covering some basic ideas would sure make things better.

So, what i’ll do today is figure out how to get your complete docker image running on an AWS GPU instance, and train my model and do the export from within your complete image. I’m guessing that will work.

4

It sounds like you had a frustrating experience, but I hope that we can help you from here. About the documentation, we do have this guide - also available in the Hailo DevZone, under Hailo ModelZoo User Guide.

As for the issue you are having, there shouldn’t be a problem with creating the network outside of our training containers, but it would indeed be easier to use them – it just ensures that all the layer names are the same, and avoids issues with start/end node names.

So you could use the ModelZoo training container, which by the way, is just a fork the ultralytics repo, along with all the compatible packages. This would make the conversion easier, and it wouldn’t be hard to do since you already went through that same training process outside of that container.

A few things I’ve noticed in your output log:

The arch is being wrongly identified as yolov8.

It’s also unclear which end-node names were used:

If you don’t want to use the training container, we can figure this out. I believe that explicitly passing the correct end-node names could be a good debugging start. Up to you which way you want to proceed.

5

Hi @George_Sexton
At DeGirum (a SW partner of Hailo), we developed a cloud compiler tool that helps users convert YOLO training checkpoints to .hef files. You can read this post for more details: Early Access to DeGirum Cloud Compiler - General - Hailo Community