I have a question regarding ONNX SpaceToDepth and DepthToSpace

General
1

According to the Dataflow Compiler documentation, it appears that both SpaceToDepth and DepthToSpace are supported.
DepthToSpace corresponds to PyTorch’s PixelShuffle, but SpaceToDepth seems to need to be implemented using tensor slice + concat.
When I looked at the ONNX graph of yolov5m in the hailo_model_zoo, it was implemented using slice + concat.

image
I configured the input part of my model as shown in the picture, but I encountered an error during parsing, as indicated in the log below.
Is there a solution to this issue?

[info] Translation started on ONNX model model
[info] Restored ONNX model model (completion time: 00:00:00.02)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.13)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:01.57)
Traceback (most recent call last):
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py”, line 220, in translate_onnx_model
parsing_results = self._parse_onnx_model_to_hn(
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py”, line 300, in _parse_onnx_model_to_hn
return self.parse_model_to_hn(
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py”, line 352, in parse_model_to_hn
hailo_nn = fuser.convert_model()
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py”, line 97, in convert_model
self._run_pre_fusing_flow()
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py”, line 122, in _run_pre_fusing_flow
self._handle_downsample_by_two_with_slice()
File “/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py”, line 4994, in _handle_downsample_by_two_with_slice
raise BackendFuserException(
hailo_sdk_client.model_translator.fuser.exceptions.BackendFuserException: base slice layer base_slice3 (translated from /Slice_1) is unsupported due to strided slicing: height_slice=[0, 540, 1], width_slice=[0, 1920, 2], features_slice=[0, 3, 1]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/media/SSD/WORKSPACE/emb/hailo/src/hailo_example.py", line 198, in <module>
    parsing(model_name)
  File "/media/SSD/WORKSPACE/emb/hailo/src/hailo_example.py", line 68, in parsing
    hn, npz = runner.translate_onnx_model(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1158, in translate_onnx_model
    parser.translate_onnx_model(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 260, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 300, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 352, in parse_model_to_hn
    hailo_nn = fuser.convert_model()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py", line 97, in convert_model
    self._run_pre_fusing_flow()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py", line 122, in _run_pre_fusing_flow
    self._handle_downsample_by_two_with_slice()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/fuser/fuser.py", line 4994, in _handle_downsample_by_two_with_slice
    raise BackendFuserException(
hailo_sdk_client.model_translator.fuser.exceptions.BackendFuserException: base slice layer base_slice3 (translated from /Slice_1) is unsupported due to strided slicing: height_slice=[0, 540, 1], width_slice=[0, 1920, 2], features_slice=[0, 3, 1]
2

Hi @dotorl127,
What model is this? from the error message it look like the connectivity of the model is not good when the Hailo Parser works on it. Is it a model that is trained on square input resolution that you’ve changed to FHD?

Regards,

3

This is custom model, input FHD and I found solution.
Anyway I have another question how apply pytorch PixelUnShuffle layer?
Is TensorFlow’s SpaceToDepth the only solution?

4

Hi @dotorl127,
Currently I believe so.
Notice that S2D is equivalent to PS for a single channels, but for RGB for example you’ll need to rearrange the output channels to get the matching outputs.

Regards,

5

image
image155×574 15.3 KB

After configuring a layer in PyTorch to output the same results as TensorFlow’s space_to_depth and depth_to_space, as shown in the picture, and then exporting it to ONNX and parsing it with the Dataflow Compiler, the following error was encountered.

[info] Translation started on ONNX model test
[info] Restored ONNX model test (completion time: 00:00:00.01)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.09)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:00.33)
Traceback (most recent call last):
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 220, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 300, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 351, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 79, in convert_model
    self._create_layers()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 34, in _create_layers
    self._add_direct_layers()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 187, in _add_direct_layers
    raise ParsingWithRecommendationException(
hailo_sdk_client.model_translator.exceptions.ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
 UnsupportedShuffleLayerError in op /Reshape: Failed to determine type of layer to create in node /Reshape
 UnsupportedShuffleLayerError in op /Transpose: Failed to determine type of layer to create in node /Transpose
 UnsupportedShuffleLayerError in op /Reshape_1: Failed to determine type of layer to create in node /Reshape_1
Please try to parse the model again, using these start node names: /Transpose

However, when I parsed the depth_to_space layer separately, it worked correctly. Why doesn’t the sequential space_to_depth layer work properly?

image

[info] Translation started on ONNX model test2
[info] Restored ONNX model test2 (completion time: 00:00:00.02)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.04)
[info] Start nodes mapped from original model: 'images': 'test2/input_layer1'.
[info] End nodes mapped from original model: '/Reshape_1'.
[info] Translation completed on ONNX model test2 (completion time: 00:00:00.16)
[info] Saved HAR to: /media/SSD/WORKSPACE/emb/hailo/models/har/test2_hailo_model.har

image

6

image

Additionally, when I replaced the reshape + transpose + reshape operations corresponding to space_to_depth with ONNX’s space_to_depth and parsed it, the results were as follows:

[info] Translation started on ONNX model test
[info] Restored ONNX model test (completion time: 00:00:00.02)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.04)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:00.20)
Traceback (most recent call last):
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 220, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 300, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 351, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 79, in convert_model
    self._create_layers()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 34, in _create_layers
    self._add_direct_layers()
  File "/home/moon/anaconda3/envs/hailo/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 187, in _add_direct_layers
    raise ParsingWithRecommendationException(
hailo_sdk_client.model_translator.exceptions.ParsingWithRecommendationException: Parsing failed. The errors found in the graph are:
 UnsupportedShuffleLayerError in op custom_added_SpaceToDepth0: Failed to determine type of layer to create in node custom_added_SpaceToDepth0
Please try to parse the model again, using these start node names: /Reshape_2
Please try to parse the model again, using these end node names: /Reshape_3

Is there a way to get the space_to_depth layer recognized when parsing a torch or ONNX model?

7

Hi @dotorl127,
The Hailo Parser supports specific operators as described in the Dataflow Compiler user guide. For specific implementation of operators as a combination, even if the layers are not supported as standalone layers, we implemented support for.
The Parser recognizes the structure of the D2S for example, and parser it accordingly. But, if the structure doesn’t fit the supported operator, like in the case you presented, the Parser doesn’t recognize it and therefore throws the unsupported operator error.

The specific S2D layer is not currently supported, but it might in the future. We do support S2D from TF (tf.nn.space_to_depth) and from tflite.

Regards,