YOLOv11 Implementation

General
, ,
1

Hi everyone,

I want to accelerate pose detection using YOLO: I’m looking for the equivalent of yolov8s_pose.hef, but with YOLO 11 or 12—so something like yolov11s_pose.hef.

Could you point me in the right direction? Based on the scripts provided on GitHub by Hailo, would it be enough to simply replace the YOLO model? That is, are the inputs and outputs of YOLO still the same across versions?

If anyone is available to suggest the necessary steps, I would really appreciate it. Alternatively, is there a repository where all YOLO 11 models are already converted into Hailo format?

Thanks in advance!

3

"I attempted to convert my custom model using hailomz compile, but the conversion does not seem to work properly. It looks like my model is not supported. Here is the command I ran and the resulting error:

(hailodfc) simolinux@btsw189:~/hailo_model_zoo$ hailomz compile --yaml ~/hailo_model_zoo/yolo11s-pose.yaml --hw-arch hailo8l yolo11s-pose
usage: hailomz compile [-h] [--yaml YAML_PATH] [--ckpt CKPT_PATH] [--hw-arch]
                       [--start-node-names START_NODE_NAMES [START_NODE_NAMES ...]]
                       [--end-node-names END_NODE_NAMES [END_NODE_NAMES ...]]
                       [--model-script MODEL_SCRIPT_PATH | --performance] [--har HAR_PATH]
                       [--calib-path CALIB_PATH] [--resize RESIZE [RESIZE ...]]
                       [--input-conversion {nv12_to_rgb,yuy2_to_rgb,rgbx_to_rgb}] [--classes]
                       [model_name]
hailomz compile: error: argument model_name: invalid choice: 'yolo11s-pose' (choose from [...])

It seems like yolo11s-pose is not recognized as a valid model. I have checked that my ONNX file exists and is correctly referenced in the YAML configuration.

Has anyone encountered this issue before? Could it be that the model is not supported, or am I missing something in the configuration? Any suggestions would be appreciated!"

4

@Simone_Tortorella
We integrated yolo11 models to our PySDK. They are available in our model zoo. You can see an example usage at: hailo_examples/examples/yolov8.ipynb at main · DeGirum/hailo_examples. You can just change the model name to yolo11n_pose--640x640_quant_hailort_multidevice_1 in the script and it should work. Please let me know if you need any further help.

1 Like
5

Hi Shashi, do you know ho to convert models at mmpose/projects/rtmpose at main · open-mmlab/mmpose · GitHub? Thpse with with 26 keypoints, such as RTMPose-m with input size 384x288? Is it possible to do that? May you help me?

6

Hi @Simone_Tortorella
We will take a look and let you know.

1 Like
7

Thank, I think that those models will be very promising on Hailo 8 (26 TOPS). But I am not able to convert them by myself.

8

Hi @Simone_Tortorella
Can you share what the problem is, so that we will know if we get stuck at same stage or if we were able to proceed further?

9

I don’t have an exact problem. I am not able to do a .onnx to . hef conversion in general, and I want that model to try them. I’ve seen many tutorials, but all the tutorials are not that clear to me :frowning:

10

In particular on jupiter I run :
runner = ClientRunner(hw_arch=chosen_hw_arch)
hn, npz = runner.translate_onnx_model(
ckpt_path,
model_name,
start_node_names=[“Conv_0”],
end_node_names=[“Mul_232”]
And I get: ---------------------------------------------------------------------------
UnsupportedModelError Traceback (most recent call last)
File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:239, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
238 try:
→ 239 parsing_results = self._parse_onnx_model_to_hn(
240 onnx_model=onnx_model,
241 net_name=valid_net_name,
242 start_node_names=start_node_names,
243 end_node_names=end_node_names,
244 net_input_shapes=net_input_shapes,
245 disable_shape_inference=disable_shape_inference,
246 net_input_format=net_input_format,
247 )
249 except Exception as e:

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:320, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
318 self._logger.warning(f"ONNX shape inference failed: {e!s}")
→ 320 return self.parse_model_to_hn(
321 onnx_model,
322 None,
323 net_name,
324 start_node_names,
325 end_node_names,
326 nn_framework=NNFramework.ONNX,
327 output_shapes=output_shapes,
328 net_input_format=net_input_format,
329 **kwargs,
330 )

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:371, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
369 raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
→ 371 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
372 hailo_nn = fuser.convert_model()

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:92, in HailoNNConverter.convert_model(self)
91 self._handle_tokens_matmul()
—> 92 self._calculate_shapes(validate_shapes=False)
93 self._add_output_layers()

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:203, in ONNXConverter._calculate_shapes(self, validate_shapes)
202 self._update_meta_graph()
→ 203 self._layers_graph.calculate_shapes(meta_edges_graph=self._meta_graph, validate_shapes=validate_shapes)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hailo_nn.py:750, in HailoNN.calculate_shapes(self, meta_edges_graph, validate_shapes)
749 for layer in graph:
→ 750 self.update_input_shapes_from_predecessors(layer)
751 if (
752 layer.op
753 in [
(…)
762 and layer.is_nv_converter()
763 ):

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hailo_nn.py:815, in HailoNN.update_input_shapes_from_predecessors(self, layer)
814 assert all(shape != for shape in input_shapes)
→ 815 layer.input_shapes = input_shapes

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hn_layers/layer.py:538, in Layer.input_shapes(self, input_shapes)
534 raise UnsupportedModelError(
535 f"Unexpected input_shapes at {self.full_name_msg}, input_shapes={input_shapes} "
536 f"(type={type(input_shapes)})",
537 )
→ 538 self.set_input_shapes(input_shapes)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hn_layers/conv2d.py:575, in Conv2DLayer.set_input_shapes(self, input_shapes, validate)
574 if input_features != self.kernel_shape[2] * calculated_groups:
→ 575 raise UnsupportedModelError(
576 f"Invalid kernel shape for {self.full_name_msg}.\nEither the input shape "
577 f"doesn’t match the kernel shape, or the calculated groups number doesn’t "
578 f"match the expected ratio between kernel shape and input shape.\n"
579 f"Kernel features: {self.kernel_shape[2]} Input features: {input_features} "
580 f"Groups: {calculated_groups}",
581 )
583 self.groups = calculated_groups

UnsupportedModelError: Invalid kernel shape for base conv layer base_conv51 (translated from MatMul_221).
Either the input shape doesn’t match the kernel shape, or the calculated groups number doesn’t match the expected ratio between kernel shape and input shape.
Kernel features: 108 Input features: 26 Groups: 0

During handling of the above exception, another exception occurred:

UnsupportedModelError Traceback (most recent call last)
Cell In[8], line 2
1 runner = ClientRunner(hw_arch=chosen_hw_arch)
----> 2 hn, npz = runner.translate_onnx_model(
3 ckpt_path,
4 model_name,
5 start_node_names=[“Conv_0”],
6 end_node_names=[“Mul_232”]
7 )

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/states/states.py:16, in allowed_states..wrap..wrapped_func(self, *args, **kwargs)
12 if self._state not in states:
13 raise InvalidStateException(
14 f"The execution of {func.name} is not available under the state: {self._state.value}",
15 )
—> 16 return func(self, *args, **kwargs)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py:1177, in ClientRunner.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
1134 “”"
1135 DFC API for parsing an ONNX model. This creates a runner with loaded HN (model) and
1136 parameters.
(…)
1174
1175 “”"
1176 parser = Parser()
→ 1177 parser.translate_onnx_model(
1178 model=model,
1179 net_name=net_name,
1180 start_node_names=start_node_names,
1181 end_node_names=end_node_names,
1182 net_input_shapes=net_input_shapes,
1183 augmented_path=augmented_path,
1184 disable_shape_inference=disable_shape_inference,
1185 disable_rt_metadata_extraction=disable_rt_metadata_extraction,
1186 net_input_format=net_input_format,
1187 **kwargs,
1188 )
1189 return self._finalize_parsing(parser.return_data)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:280, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
277 milestone = self._format_time_milestone(start_time)
278 self._logger.info(f"Simplified ONNX model for a parsing retry attempt (completion time: {milestone})“)
→ 280 parsing_results = self._parse_onnx_model_to_hn(
281 onnx_model=simplified_model,
282 net_name=valid_net_name,
283 start_node_names=start_node_names,
284 end_node_names=end_node_names,
285 net_input_shapes=net_input_shapes,
286 disable_shape_inference=disable_shape_inference,
287 net_input_format=net_input_format,
288 **kwargs,
289 )
291 milestone = self._format_time_milestone(start_time)
292 self._logger.info(f"Translation completed on ONNX model {valid_net_name} (completion time: {milestone})”)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:320, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
317 except Exception as e:
318 self._logger.warning(f"ONNX shape inference failed: {e!s}")
→ 320 return self.parse_model_to_hn(
321 onnx_model,
322 None,
323 net_name,
324 start_node_names,
325 end_node_names,
326 nn_framework=NNFramework.ONNX,
327 output_shapes=output_shapes,
328 net_input_format=net_input_format,
329 **kwargs,
330 )

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:371, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
368 else:
369 raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
→ 371 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
372 hailo_nn = fuser.convert_model()
373 hailo_nn.validate_stage(HnStage.HN)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:92, in HailoNNConverter.convert_model(self)
90 self._handle_inner_product_matmul()
91 self._handle_tokens_matmul()
—> 92 self._calculate_shapes(validate_shapes=False)
93 self._add_output_layers()
94 self._handle_fused_layers()

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:203, in ONNXConverter._calculate_shapes(self, validate_shapes)
201 self._update_attention_windows()
202 self._update_meta_graph()
→ 203 self._layers_graph.calculate_shapes(meta_edges_graph=self._meta_graph, validate_shapes=validate_shapes)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hailo_nn.py:750, in HailoNN.calculate_shapes(self, meta_edges_graph, validate_shapes)
748 graph = nx.lexicographical_topological_sort(meta_edges_graph) if meta_edges_graph else self.stable_toposort()
749 for layer in graph:
→ 750 self.update_input_shapes_from_predecessors(layer)
751 if (
752 layer.op
753 in [
(…)
762 and layer.is_nv_converter()
763 ):
764 layer.output_copies = 1

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hailo_nn.py:815, in HailoNN.update_input_shapes_from_predecessors(self, layer)
813 self.add_edge(pred, layer, out_shape=pred_output_shape, in_shape=layer.reshape_input(pred_output_shape))
814 assert all(shape != for shape in input_shapes)
→ 815 layer.input_shapes = input_shapes

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hn_layers/layer.py:538, in Layer.input_shapes(self, input_shapes)
533 elif any(not isinstance(shape, list) for shape in input_shapes):
534 raise UnsupportedModelError(
535 f"Unexpected input_shapes at {self.full_name_msg}, input_shapes={input_shapes} "
536 f"(type={type(input_shapes)})",
537 )
→ 538 self.set_input_shapes(input_shapes)

File ~/hailodfc/lib/python3.10/site-packages/hailo_sdk_common/hailo_nn/hn_layers/conv2d.py:575, in Conv2DLayer.set_input_shapes(self, input_shapes, validate)
571 # We don’t validate that the feature dimensions are equal in case of conv with ew_add.
572 # In that case, there may be two different input_features (one for each input).
573 # The input features are validated in _calc_conv_ew_add_output_shape.
574 if input_features != self.kernel_shape[2] * calculated_groups:
→ 575 raise UnsupportedModelError(
576 f"Invalid kernel shape for {self.full_name_msg}.\nEither the input shape "
577 f"doesn’t match the kernel shape, or the calculated groups number doesn’t "
578 f"match the expected ratio between kernel shape and input shape.\n"
579 f"Kernel features: {self.kernel_shape[2]} Input features: {input_features} "
580 f"Groups: {calculated_groups}",
581 )
583 self.groups = calculated_groups

UnsupportedModelError: Invalid kernel shape for base conv layer base_conv51 (translated from MatMul_221).
Either the input shape doesn’t match the kernel shape, or the calculated groups number doesn’t match the expected ratio between kernel shape and input shape.
Kernel features: 108 Input features: 26 Groups: 0.

11

Hi @Simone_Tortorella
Our team took a look at this model and our unfortunate conclusion is that it needs too many modifications to compile to a hef file. We can try to train a yolo-pose with 26 checkpoints and compare its accuracy with this model to see if a yolo based model can be used. Let me know what you think.

12

Yes, I think you can give it a try. Halpe 26 sould be the standard model, so it does need feet.