Dataflow compiler parsing failure - incorrectly reshapes array

General
1

Hi there,

I am evaluating the Hailo as an option for an on-device inference accelerator. I have a model that was trained using Pytorch and compiled to ONNX. When attempting to convert the ONNX file to HAR using the Dataflow Compiler, it raises an error where it incorrectly tries to shape an array:

ValueError: cannot reshape array of size 12513 into shape (1,129,1)

The above should be (1, 129, 97). Previously, this model was successfully compiled using Intel’s OpenVINO (compiled for inference on their Neural Compute Stick 2).

I am happy to share the .onnx file if there is a way to do so. I’ve inspected the the onnx graph using Netron (https://netron.app/) but I haven’t been able to spot where the issue arises.

Any help or advice is much appreciated, thanks! The full verbose traceback is below:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:239, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
    238 try:
--> 239     parsing_results = self._parse_onnx_model_to_hn(
    240         onnx_model=onnx_model,
    241         net_name=valid_net_name,
    242         start_node_names=start_node_names,
    243         end_node_names=end_node_names,
    244         net_input_shapes=net_input_shapes,
    245         disable_shape_inference=disable_shape_inference,
    246         net_input_format=net_input_format,
    247     )
    249 except Exception as e:

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:320, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
    318         self._logger.warning(f"ONNX shape inference failed: {e!s}")
--> 320 return self.parse_model_to_hn(
    321     onnx_model,
    322     None,
    323     net_name,
    324     start_node_names,
    325     end_node_names,
    326     nn_framework=NNFramework.ONNX,
    327     output_shapes=output_shapes,
    328     net_input_format=net_input_format,
    329     **kwargs,
    330 )

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:371, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
    369     raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
--> 371 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
    372 hailo_nn = fuser.convert_model()

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:83, in HailoNNConverter.convert_model(self)
     82 self._validate_bn_ops_in_training()
---> 83 self._create_layers()
     84 self._add_layers_connections()

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:39, in EdgeNNConverter._create_layers(self)
     38 self._update_vertices_info()
---> 39 self._add_direct_layers()
     40 self._validate_processed_vertices()

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:121, in EdgeNNConverter._add_direct_layers(self)
    120 self._logger.debug(f"Processing vertex {vertex.name}")
--> 121 self._layer_callback_from_vertex(vertex)
    122 self._visited_states[vertex] = VertexState.PROCESSED

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:430, in ONNXConverter._layer_callback_from_vertex(self, vertex)
    429 if vertex.is_ew_add():
--> 430     self._create_ew_add_layer(vertex)
    431 elif vertex.is_mul_by_2_ew_add():

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1158, in ONNXConverter._create_ew_add_layer(self, vertex)
   1157 def _create_ew_add_layer(self, vertex):
-> 1158     self._create_ew_layer(vertex, EWAddLayer)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1181, in ONNXConverter._create_ew_layer(self, vertex, layer_cls)
   1180 if is_ew_op_with_const_input:
-> 1181     const_layer, ew_op_input = self._create_ew_const_input_layer(vertex)
   1183 layer = layer_cls.create(
   1184     vertex.name,
   1185     ew_op_input,
   1186     output_shapes=vertex.get_output_shapes(validate_zero_dims=True),
   1187 )

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1213, in ONNXConverter._create_ew_const_input_layer(self, vertex)
   1212         const_output_shape = [-1, ew_op_output_shape[1], ew_op_output_shape[3], ew_op_output_shape[2]]
-> 1213         input_values = np.reshape(input_values, [1, *ew_op_output_shape[2:]])
   1214 else:

File <__array_function__ internals>:180, in reshape(*args, **kwargs)

File ~/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py:298, in reshape(a, newshape, order)
    200 """
    201 Gives a new shape to an array without changing its data.
    202 
   (...)
    296        [5, 6]])
    297 """
--> 298 return _wrapfunc(a, 'reshape', newshape, order=order)

File ~/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py:57, in _wrapfunc(obj, method, *args, **kwds)
     56 try:
---> 57     return bound(*args, **kwds)
     58 except TypeError:
     59     # A TypeError occurs if the object does have such a method in its
     60     # class, but its signature is not identical to that of NumPy's. This
   (...)
     64     # Call _wrapit from within the except clause to ensure a potential
     65     # exception has a traceback chain.

ValueError: cannot reshape array of size 12513 into shape (1,129,1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[42], line 2
      1 runner = ClientRunner(hw_arch=chosen_hw_arch)
----> 2 hn, npz = runner.translate_onnx_model(
      3     onnx_path,
      4     onnx_model_name,
      5     # disable_shape_inference=True,
      6     # start_node_names=["onnx::Cast_0"],
      7     # end_node_names=["192"],
      8     # net_input_shapes={"onnx::Cast_0": [1, 1, 193, 1032]},
      9 )

File ~/.local/lib/python3.10/site-packages/hailo_sdk_common/states/states.py:16, in allowed_states.<locals>.wrap.<locals>.wrapped_func(self, *args, **kwargs)
     12 if self._state not in states:
     13     raise InvalidStateException(
     14         f"The execution of {func.__name__} is not available under the state: {self._state.value}",
     15     )
---> 16 return func(self, *args, **kwargs)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py:1177, in ClientRunner.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
   1134 """
   1135 DFC API for parsing an ONNX model. This creates a runner with loaded HN (model) and
   1136 parameters.
   (...)
   1174 
   1175 """
   1176 parser = Parser()
-> 1177 parser.translate_onnx_model(
   1178     model=model,
   1179     net_name=net_name,
   1180     start_node_names=start_node_names,
   1181     end_node_names=end_node_names,
   1182     net_input_shapes=net_input_shapes,
   1183     augmented_path=augmented_path,
   1184     disable_shape_inference=disable_shape_inference,
   1185     disable_rt_metadata_extraction=disable_rt_metadata_extraction,
   1186     net_input_format=net_input_format,
   1187     **kwargs,
   1188 )
   1189 return self._finalize_parsing(parser.return_data)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:280, in Parser.translate_onnx_model(self, model, net_name, start_node_names, end_node_names, net_input_shapes, augmented_path, disable_shape_inference, disable_rt_metadata_extraction, net_input_format, **kwargs)
    277     milestone = self._format_time_milestone(start_time)
    278     self._logger.info(f"Simplified ONNX model for a parsing retry attempt (completion time: {milestone})")
--> 280     parsing_results = self._parse_onnx_model_to_hn(
    281         onnx_model=simplified_model,
    282         net_name=valid_net_name,
    283         start_node_names=start_node_names,
    284         end_node_names=end_node_names,
    285         net_input_shapes=net_input_shapes,
    286         disable_shape_inference=disable_shape_inference,
    287         net_input_format=net_input_format,
    288         **kwargs,
    289     )
    291 milestone = self._format_time_milestone(start_time)
    292 self._logger.info(f"Translation completed on ONNX model {valid_net_name} (completion time: {milestone})")

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:320, in Parser._parse_onnx_model_to_hn(self, onnx_model, net_name, start_node_names, end_node_names, net_input_shapes, disable_shape_inference, net_input_format, **kwargs)
    317     except Exception as e:
    318         self._logger.warning(f"ONNX shape inference failed: {e!s}")
--> 320 return self.parse_model_to_hn(
    321     onnx_model,
    322     None,
    323     net_name,
    324     start_node_names,
    325     end_node_names,
    326     nn_framework=NNFramework.ONNX,
    327     output_shapes=output_shapes,
    328     net_input_format=net_input_format,
    329     **kwargs,
    330 )

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/parser/parser.py:371, in Parser.parse_model_to_hn(self, model, values, net_name, start_node_names, end_node_names, nn_framework, output_shapes, net_input_format, rename_layers_by_blocks)
    368 else:
    369     raise BackendRuntimeException(f"Unsupported NN framework {nn_framework}")
--> 371 fuser = HailoNNFuser(converter.convert_model(), net_name, converter.end_node_names)
    372 hailo_nn = fuser.convert_model()
    373 hailo_nn.validate_stage(HnStage.HN)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/translator.py:83, in HailoNNConverter.convert_model(self)
     81 self._validate_model_params()
     82 self._validate_bn_ops_in_training()
---> 83 self._create_layers()
     84 self._add_layers_connections()
     85 self._layers_graph.set_names_and_indices()

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:39, in EdgeNNConverter._create_layers(self)
     37 self._add_input_layers()
     38 self._update_vertices_info()
---> 39 self._add_direct_layers()
     40 self._validate_processed_vertices()

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/edge_nn_translator.py:121, in EdgeNNConverter._add_direct_layers(self)
    119 elif vertex not in self._visited_states:
    120     self._logger.debug(f"Processing vertex {vertex.name}")
--> 121     self._layer_callback_from_vertex(vertex)
    122     self._visited_states[vertex] = VertexState.PROCESSED
    124 for node in sorted(self._graph.successors(vertex), key=attrgetter("name")):

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:430, in ONNXConverter._layer_callback_from_vertex(self, vertex)
    428 elif vertex.op in ADD_OPS:
    429     if vertex.is_ew_add():
--> 430         self._create_ew_add_layer(vertex)
    431     elif vertex.is_mul_by_2_ew_add():
    432         self._create_normalization_layer(vertex)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1158, in ONNXConverter._create_ew_add_layer(self, vertex)
   1157 def _create_ew_add_layer(self, vertex):
-> 1158     self._create_ew_layer(vertex, EWAddLayer)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1181, in ONNXConverter._create_ew_layer(self, vertex, layer_cls)
   1179 const_layer = None
   1180 if is_ew_op_with_const_input:
-> 1181     const_layer, ew_op_input = self._create_ew_const_input_layer(vertex)
   1183 layer = layer_cls.create(
   1184     vertex.name,
   1185     ew_op_input,
   1186     output_shapes=vertex.get_output_shapes(validate_zero_dims=True),
   1187 )
   1188 self._add_layer(layer)

File ~/.local/lib/python3.10/site-packages/hailo_sdk_client/model_translator/onnx_translator/onnx_translator.py:1213, in ONNXConverter._create_ew_const_input_layer(self, vertex)
   1211     else:
   1212         const_output_shape = [-1, ew_op_output_shape[1], ew_op_output_shape[3], ew_op_output_shape[2]]
-> 1213         input_values = np.reshape(input_values, [1, *ew_op_output_shape[2:]])
   1214 else:
   1215     const_output_shape = [-1, *ew_op_output_shape[1:4]]

File <__array_function__ internals>:180, in reshape(*args, **kwargs)

File ~/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py:298, in reshape(a, newshape, order)
    198 @array_function_dispatch(_reshape_dispatcher)
    199 def reshape(a, newshape, order='C'):
    200     """
    201     Gives a new shape to an array without changing its data.
    202 
   (...)
    296            [5, 6]])
    297     """
--> 298     return _wrapfunc(a, 'reshape', newshape, order=order)

File ~/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py:57, in _wrapfunc(obj, method, *args, **kwds)
     54     return _wrapit(obj, method, *args, **kwds)
     56 try:
---> 57     return bound(*args, **kwds)
     58 except TypeError:
     59     # A TypeError occurs if the object does have such a method in its
     60     # class, but its signature is not identical to that of NumPy's. This
   (...)
     64     # Call _wrapit from within the except clause to ensure a potential
     65     # exception has a traceback chain.
     66     return _wrapit(obj, method, *args, **kwds)

ValueError: cannot reshape array of size 12513 into shape (1,129,1)
2

Hey @i_j ,

ValueError: cannot reshape array of size 12513 into shape (1,129,1)

This error occurs when the Hailo Dataflow Compiler tries to handle element-wise operations with constant tensors, particularly when it attempts to reshape a tensor but encounters a size mismatch.

The compiler is trying to reshape a constant tensor of size 12513 into shape (1,129,1), but it should be (1,129,97). This is a known edge case in Hailo’s compiler when handling:

  • Element-wise operations with constant tensors
  • Broadcasting mismatches between ONNX dynamic behavior and Hailo’s static requirements
  • Audio or spectrogram-based models (which commonly have these patterns)

Recommended Solutions

1. Manually Specify Input Shapes

runner.translate_onnx_model(
    model="model.onnx",
    net_name="my_model",
    net_input_shapes={"input_tensor_name": [1, 129, 97]}
)

2. Disable Shape Inference

runner.translate_onnx_model(
    model="model.onnx",
    net_name="my_model",
    disable_shape_inference=True
)

3. Use Custom Start/End Points

runner.translate_onnx_model(
    model="model.onnx",
    net_name="my_model",
    start_node_names=["input_tensor"],
    end_node_names=["output_tensor"]
)

4. Check Model Compatibility First

hailomz parse model.onnx --target <chip> --net-input-shape [1,129,97]

Additional Approach

You also need to edit the ONNX file to reshape the problematic constant node (size 12513) to the correct target shape using tools like onnx-simplifier or Python with onnx.helper.

4

Hi @omria, thanks for providing a few things to try!

I gave each of these a whirl but without success, unfortunately.

The input image shape is (1, 772, 1032) which after a few layers, becomes the grid of (1, 97, 129). onnx-simplifier produced a graph that also did not compile with the Dataflow Compiler.

Is there some way for me to send you folks the onnx file I am working with?

5

Hi, just wanted to follow up - is there a way to share a .onnx file with the dev team? I’d love to get this to compile and it could possibly be useful to the team to have an edge case to test on.

@omria