Dataflow Compiler has been crashed when parsing a concat-conv layer

[kuki]

Hello,

When parsing my own onnx model, it crashed in a concat-conv layer. Models are trained on PyTorch 2.1.2 and converted to ONNX format on PyTorch. The Dataflow Compiler is 3.29. Is it possible to parse such a layer?

The log is as follows.

root@01ad0936a61c:/# uname -a
Linux 01ad0936a61c 5.15.161+ #1 SMP Fri Sep 27 10:32:46 UTC 2024 x86_64 GNU/Linux

root@01ad0936a61c:/# hailo parser onnx --hw-arch hailo8l ./model.onnx --start-node-names onnx::Gather_0 --end-node-names 1293
[info] Current Time: 01:39:08, 10/02/24
[info] CPU: Architecture: , Model: Intel(R) Xeon(R) CPU @ 2.20GHz, Number Of Cores: 2, Utilization: 0.5%
[info] Memory: Total: 7GB, Available: 7GB
[info] System info: OS: Linux, Kernel: 5.15.161+
[info] Hailo DFC Version: 3.29.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo parser onnx --hw-arch hailo8l ./model.onnx --start-node-names onnx::Gather_0 --end-node-names 1293`
[info] Translation started on ONNX model trash_detection
[info] Restored ONNX model trash_detection (completion time: 00:00:01.13)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:07.14)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:15.49)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 235, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 316, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 367, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 83, in convert_model
    self._create_layers()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 40, in _create_layers
    self._add_direct_layers()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 122, in _add_direct_layers
    self._layer_callback_from_vertex(vertex)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 290, in _layer_callback_from_vertex
    consumed_vertices = self._create_convolutional_layer(vertex)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 658, in _create_convolutional_layer
    kernel, bias = vertex.get_conv_over_groups_info()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 4454, in get_conv_over_groups_info
    features = self.get_output_shapes(convert_to_nhwc=False)[0][self.output_format.index(Dims.CHANNELS)]
ValueError: channels is not in list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/hailo", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/main.py", line 111, in main
    ret_val = client_command_runner.run()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 68, in run
    return self._run(argv)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py", line 89, in _run
    return args.func(args)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/tools/parser_cli.py", line 198, in run
    runner = self._parse(net_name, args, tensor_shapes)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/tools/parser_cli.py", line 282, in _parse
    runner.translate_onnx_model(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_common/states/states.py", line 16, in wrapped_func
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py", line 1158, in translate_onnx_model
    parser.translate_onnx_model(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 276, in translate_onnx_model
    parsing_results = self._parse_onnx_model_to_hn(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 316, in _parse_onnx_model_to_hn
    return self.parse_model_to_hn(
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py", line 367, in parse_model_to_hn
    fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py", line 83, in convert_model
    self._create_layers()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 40, in _create_layers
    self._add_direct_layers()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py", line 122, in _add_direct_layers
    self._layer_callback_from_vertex(vertex)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 290, in _layer_callback_from_vertex
    consumed_vertices = self._create_convolutional_layer(vertex)
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py", line 658, in _create_convolutional_layer
    kernel, bias = vertex.get_conv_over_groups_info()
  File "/usr/local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_graph.py", line 4454, in get_conv_over_groups_info
    features = self.get_output_shapes(convert_to_nhwc=False)[0][self.output_format.index(Dims.CHANNELS)]
ValueError: channels is not in list

The corresponding layer on the model is shown in the figure 1.

If the shape of input was fixed, the same error occurred at the same point.

[omria]

Hey @kuki

Welcome to the Hailo Community!

The parsing issue with your ONNX model’s concat-conv layer may be due to how the Hailo parser handles layers during translation. Consider these points:

1. Verify that your model’s layers are supported by the current Hailo AI software suite version.
2. Check the channel format. The parser may expect a specific format like NCHW or NHWC.
3. Try customizing the parsing process using start and end node names or by simplifying the model.
4. Ensure you’re using the correct YAML configuration from the Hailo Model Zoo for your model.

If these steps don’t resolve the issue, consider contacting Hailo support with detailed layer configurations and error logs.

[kuki]

Thanks for the reply!

We verified item 3. by making the start and end node part of the model. As a result, We were able to successfully compile the model by simplifying the dimensionality reduction by Gather in the model input layer as shown in the attached figure.

We think that after passing through the Gather op, the channel information was not propagated to the layer behind, resulting in the error ValueError: channels are not in list.

Fig 1. Before removing the Gather op.

Fig 2. After removing the Gather op.