Hey there,
I would like to run Megadetector on Hailo8L.
The weights are available as a .pt file that can be converted to .onnx.
The model is based on yolov5x6, which is not in the model zoo.
Is there any way to compile it using the tools within the software suite?
Yes, once you have an ONNX you can run it through the Dataflow Compiler pipeline to convert it into a HEF file. You can follow the instructions on how to do it using the Dataflow Compiler user guide in the Hailo website under Developer Zone → Documentation.
You can also run, while inside the Suite docker, the CLI command “hailo -h” and it will give you a short description of the Dataflow Compiler’s pipeline.
Regards,
Dear Omer,
thanks for your input. For now I am focussing on Megadetector v6c that is based on YOLOv9c
I was trying to follow the tutorial but the code fails in the first couple of lines.
docker hailo software suite v2024-7.1
The first steps of the example code (working fine)
import tensorflow as tf
from IPython.display import SVG
from hailo_sdk_client import ClientRunner
chosen_hw_arch = "hailo8l"
onnx_model_name = "resnet_v1_18"
onnx_path = "/local/workspace/shared_with_docker/resnet_v1_18.onnx"
runner = ClientRunner(hw_arch=chosen_hw_arch)
hn, npz = runner.translate_onnx_model(
onnx_path,
onnx_model_name,
start_node_names=["input.1"],
end_node_names=["191"],
net_input_shapes={"input.1": [1, 3, 224, 224]},
)
The modified code with a model based on yolov9c architecture:
onnx_model_name = "mdv6"
onnx_path = "/local/workspace/shared_with_docker/MDV6b-yolov9c.onnx"
runner = ClientRunner(hw_arch=chosen_hw_arch)
hn, npz = runner.translate_onnx_model(
onnx_path,
onnx_model_name,
start_node_names=["images"],
end_node_names=["output0"],
net_input_shapes={"images": [1,3,640,640]},
)
results in below output and subsequent error chain:
[info] Translation started on ONNX model mdv6
[info] Restored ONNX model mdv6 (completion time: 00:00:00.61)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:02.48)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:09.45)
{
"name": "ParsingWithRecommendationException",
"message": "Parsing failed. The errors found in the graph are:
UnsupportedSliceLayerError in op /model.22/Slice: Failed to create slice layer at vertex /model.22/Slice. Slice on axis 1 is not supported
UnsupportedSliceLayerError in op /model.22/Slice_1: Failed to create slice layer at vertex /model.22/Slice_1. Slice on axis 1 is not supported
UnsupportedModelError in op /model.22/Sub: In vertex /model.22/Sub_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
UnsupportedModelError in op /model.22/Add_1: In vertex /model.22/Add_1_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
Please try to parse the model again, using these end node names: /model.22/dfl/Reshape_1, /model.22/Sigmoid",
"stack": "---------------------------------------------------------------------------
ParsingWithRecommendationException Traceback (most recent call last)
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:220, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
219 try:
--> 220 parsing_results = self._parse_onnx_model_to_hn(
221 onnx_model=onnx_model,
222 net_name=valid_net_name,
223 start_node_names=start_node_names,
224 end_node_names=end_node_names,
225 net_input_shapes=net_input_shapes,
226 disable_shape_inference=disable_shape_inference,
227 net_input_format=net_input_format,
228 )
230 except Exception as e:
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:300, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
298 self._logger.warning(f\"ONNX shape inference failed: {e!s}\")
--> 300 return self.parse_model_to_hn(
301 onnx_model,
302 None,
303 net_name,
304 start_node_names,
305 end_node_names,
306 nn_framework=NNFramework.ONNX,
307 output_shapes=output_shapes,
308 net_input_format=net_input_format,
309 **kwargs,
310 )
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:351, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
349 raise BackendRuntimeException(f\"Unsupported NN framework {nn_framework}\")
--> 351 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
352 hailo_nn = fuser.convert_model()
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/translator.py:79, in HailoNNConverter.convert_model(self)
78 self._validate_bn_ops_in_training()
---> 79 self._create_layers()
80 self._add_layers_connections()
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:34, in EdgeNNConverter._create_layers(self)
33 self._update_vertices_info()
---> 34 self._add_direct_layers()
35 self._validate_processed_vertices()
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:187, in EdgeNNConverter._add_direct_layers(self)
186 if start_suggestions or end_suggestions:
--> 187 raise ParsingWithRecommendationException(
188 suggestions_msg,
189 recommended_start_node_names=list(set(start_suggestions)),
190 recommended_end_node_names=list(set(end_suggestions)),
191 parsing_report=self.get_parsing_report(from_error=True),
192 )
ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
UnsupportedSliceLayerError in op /model.22/Slice: Failed to create slice layer at vertex /model.22/Slice. Slice on axis 1 is not supported
UnsupportedSliceLayerError in op /model.22/Slice_1: Failed to create slice layer at vertex /model.22/Slice_1. Slice on axis 1 is not supported
UnsupportedModelError in op /model.22/Sub: In vertex /model.22/Sub_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
UnsupportedModelError in op /model.22/Add_1: In vertex /model.22/Add_1_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
Please try to parse the model again, using these end node names: /model.22/Mul_1, /model.22/Mul, /model.22/dfl/Reshape_1, /model.22/Sigmoid
During handling of the above exception, another exception occurred:
ParsingWithRecommendationException Traceback (most recent call last)
Cell In [4], line 4
2 onnx_path = \"/local/workspace/shared_with_docker/MDV6b-yolov9c.onnx\"
3 runner = ClientRunner(hw_arch=chosen_hw_arch)
----> 4 hn, npz = runner.translate_onnx_model(
5 onnx_path,
6 onnx_model_name,
7 start_node_names=[\"images\"],
8 end_node_names=[\"output0\"],
9 net_input_shapes={\"images\": [1,3,640,640]},
10 )
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_common/states/states.py:16, in allowed_states.<locals>.wrap.<locals>.wrapped_func(self, *args, **kwargs)
12 if self._state not in states:
13 raise InvalidStateException(
14 f\"The execution of {func.__name__} is not available under the state: {self._state.value}\",
15 )
---> 16 return func(self, *args, **kwargs)
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/runner/client_runner.py:1158, in ClientRunner.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
1115 \"\"\"
1116 DFC API for parsing an ONNX model. This creates a runner with loaded HN (model) and
1117 parameters.
(...)
1155
1156 \"\"\"
1157 parser = Parser()
-> 1158 parser.translate_onnx_model(
1159 model=model,
1160 net_name=net_name,
1161 start_node_names=start_node_names,
1162 end_node_names=end_node_names,
1163 net_input_shapes=net_input_shapes,
1164 augmented_path=augmented_path,
1165 disable_shape_inference=disable_shape_inference,
1166 disable_rt_metadata_extraction=disable_rt_metadata_extraction,
1167 net_input_format=net_input_format,
1168 **kwargs,
1169 )
1171 return self._finalize_parsing(parser.return_data)
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:260, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
257 milestone = self._format_time_milestone(start_time)
258 self._logger.info(f\"Simplified ONNX model for a parsing retry attempt (completion time: {milestone})\")
--> 260 parsing_results = self._parse_onnx_model_to_hn(
261 onnx_model=simplified_model,
262 net_name=valid_net_name,
263 start_node_names=start_node_names,
264 end_node_names=end_node_names,
265 net_input_shapes=net_input_shapes,
266 disable_shape_inference=disable_shape_inference,
267 net_input_format=net_input_format,
268 **kwargs,
269 )
271 milestone = self._format_time_milestone(start_time)
272 self._logger.info(f\"Translation completed on ONNX model {valid_net_name} (completion time: {milestone})\")
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:300, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
297 except Exception as e:
298 self._logger.warning(f\"ONNX shape inference failed: {e!s}\")
--> 300 return self.parse_model_to_hn(
301 onnx_model,
302 None,
303 net_name,
304 start_node_names,
305 end_node_names,
306 nn_framework=NNFramework.ONNX,
307 output_shapes=output_shapes,
308 net_input_format=net_input_format,
309 **kwargs,
310 )
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:351, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
348 else:
349 raise BackendRuntimeException(f\"Unsupported NN framework {nn_framework}\")
--> 351 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
352 hailo_nn = fuser.convert_model()
353 hailo_nn.validate_stage(HnStage.HN)
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/translator.py:79, in HailoNNConverter.convert_model(self)
77 self._validate_model_params()
78 self._validate_bn_ops_in_training()
---> 79 self._create_layers()
80 self._add_layers_connections()
81 self._layers_graph.set_names_and_indices()
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:34, in EdgeNNConverter._create_layers(self)
32 self._add_input_layers()
33 self._update_vertices_info()
---> 34 self._add_direct_layers()
35 self._validate_processed_vertices()
File /local/workspace/hailo_virtualenv/lib/python3.8/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:187, in EdgeNNConverter._add_direct_layers(self)
185 suggestions_msg = \"More possible end nodes found, consider using them for more precise parsing\"
186 if start_suggestions or end_suggestions:
--> 187 raise ParsingWithRecommendationException(
188 suggestions_msg,
189 recommended_start_node_names=list(set(start_suggestions)),
190 recommended_end_node_names=list(set(end_suggestions)),
191 parsing_report=self.get_parsing_report(from_error=True),
192 )
ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
UnsupportedSliceLayerError in op /model.22/Slice: Failed to create slice layer at vertex /model.22/Slice. Slice on axis 1 is not supported
UnsupportedSliceLayerError in op /model.22/Slice_1: Failed to create slice layer at vertex /model.22/Slice_1. Slice on axis 1 is not supported
UnsupportedModelError in op /model.22/Sub: In vertex /model.22/Sub_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
UnsupportedModelError in op /model.22/Add_1: In vertex /model.22/Add_1_input the contstant value shape (1, 2, 8400) must be broadcastable to the output shape [1, 8400, 2]
Please try to parse the model again, using these end node names: /model.22/dfl/Reshape_1, /model.22/Sigmoid"
}
Seems like at some point he shapes of the tensors do not match.
Any idea how to fix it?
Hi @stephan.muekusch,
The error states that there are some unsupported operations in the model that the Hailo Parser doesn’t support.
If you look at the end of the error message, it gives you the suggestion of what end nodes you should provide.
As a rule of thumb, the end nodes for the parser should be the last neural layers of the model, e.g. the last layers before the postprocessing of the model starts.
Regards,
Changing the end layers to the suggested ones brought me some steps further.
Thanks for your hint, Omer.
Now, I got stuck at compiling the actual .hef file:
runner = ClientRunner(har=quantized_model_har_path)
hef = runner.compile(
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 56s)
e[?25h
[error] Failed to produce compiled graph
No successful assignment for: format_conversion1, concat36, feature_splitter6, shortcut_softmax1, reduce_max_softmax1, ew_sub_softmax1, reduce_sum_softmax1, ew_mult_softmax1, conv145
Do you know, why it fails?
Hi @stephan.muekusch,
It seems that there are specific layers that the allocator doesn’t find a possible solution for. I can be caused from a missing feature or in the Hailo Compiler or a bug. It will be hard for me to know without seeing the model or at least getting some info on the layers it fails on.
Can you provide some details about the Softmax layer and conv145?
Regards,
Hey, I was trying to compile a model from ONNX to .hef but the compilation ends in an error:
! hailo compiler /content/yolo5m_wo_spp_to_compile.har
[info] System info: OS: Linux, Kernel: 6.1.85+
[info] Hailo DFC Version: 3.27.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler /content/yolo5m_wo_spp_to_compile.har`
[info] Compiling network
[error] Failed to produce compiled graph
[error] TypeError: expected str, bytes or os.PathLike object, not NoneType
Hi @shivatejasirimalla,
Please open a different topic for this issue.
Regards,
Hi @stephan.muekusch,
I suspect that the proposed end node names by the Parser are not good. Can you please send me the ONNX itself so I can take a look?
Regards,
Sure. I added the onnx file to the shared folder. Link via DM.
Best regards, Stephan
Hi @stephan.muekusch,
As I suspected, the issue was that the parser suggestion were not good.
The end nodes you need to provide are these:
/model.22/cv2.0/cv2.0.2/Conv
/model.22/cv3.0/cv3.0.2/Conv
/model.22/cv2.1/cv2.1.2/Conv
/model.22/cv3.1/cv3.1.2/Conv
/model.22/cv2.2/cv2.2.2/Conv
/model.22/cv3.2/cv3.2.2/Conv
If you open the ONNX in Netron, you can see that the end nodes I chose are the ones just before the start of the postprocessing ops. For example:
Once you provide the Parser these end nodes, you’ll see that it would work (you can also add Hailo NMS to this model. Please see “nms_postprocess” in the Dataflow Compiler user guide and the relevant yolov9 .yaml in GitHub - hailo-ai/hailo_model_zoo: The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment).
Regards,
