[error] Failed to produce compiled graph

sslee1 · March 24, 2025, 8:46am

I’m using the Hailo tutorial to convert my custom model.
I’ve successfully completed the steps from ONNX → HAR → Quantization HAR, but I’m getting the following error during the hef = runner.compile() step:

It seems that the input sizes for ew_mult1 are mismatched, which might be causing the issue.

How can I fix this?

[error] Failed to produce compiled graph
BackendAllocatorException: Compilation failed: The graph is invalid. The input shapes of node ew_mult1 are not aligned (1, 16384, 24) != (128, 128, 24)

sslee1 · April 1, 2025, 6:27am

Is there any guide for this issue?

omria · April 1, 2025, 10:54am

Hey @sslee1 ,

The issue arises from a tensor shape mismatch during an ElementWise operation—specifically at the node ew_mult1 during compilation.

From your model graph, I can see that ew_mult1 receives two inputs:

One input has the shape [-1, 1, 16384, 24]
The other, coming from a resize operation, has shape [-1, 1, 1, 24]

This smaller tensor is likely being broadcasted implicitly in your original framework (TensorFlow or PyTorch), which is supported by those runtimes. However:

The DFC does not support automatic broadcasting in all scenarios, particularly when spatial dimensions differ. It requires explicit shape alignment between tensors in elementwise operations.

To resolve this, you should explicitly reshape or tile the [1, 1, 24] tensor to match [1, 16384, 24] before the ew_mult1 operation.

Here’s how to fix it in your model before exporting to ONNX:

For TensorFlow:

# If tensor shape is [B, 1, 1, 24]
x = tf.tile(x, [1, 1, 16384, 1])  # Shape becomes [B, 1, 16384, 24]

For PyTorch:

# If tensor shape is [B, 1, 1, 24]
x = x.expand(-1, 1, 16384, -1)    # Or use repeat: x = x.repeat(1, 1, 16384, 1)

Make sure this reshape aligns with your model’s logic (e.g. applying a per-channel scale across spatial positions).

You should re exporting the ONNX and re Parsing you’re model.
This should allow the compiler to proceed without triggering shape mismatch errors.

Yuri_Brigance · September 26, 2025, 9:48pm

I’ve added x.expand() in PyTorch and I do see the expansion layer in the ONNX model:

However, after parsing, the resulting .har file has this layer removed:

Resulting in a similar error:

```
Traceback (most recent call last):
File “/home/yuri/Source/hailo/hailo/bin/hailo”, line 7, in
sys.exit(main())
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/main.py”, line 111, in main
ret_val = client_command_runner.run()
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py”, line 68, in run
return self._run(argv)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/cmd_utils/base_utils.py”, line 89, in _run
return args.func(args)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/tools/optimize_cli.py”, line 120, in run
self._runner.optimize(dataset, work_dir=args.work_dir)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2206, in optimize
result = self._optimize(
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 2025, in _optimize
checkpoint_info = self._sdk_backend.full_quantization(
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1138, in full_quantization
new_checkpoint_info = self._full_acceleras_run(
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1376, in _full_acceleras_run
new_checkpoint_info = self._optimization_flow_runner(optimization_flow, checkpoint_info)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 2029, in _optimization_flow_runner
optimization_flow.run()
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/orchestator.py”, line 239, in wrapper
return func(self, *args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 357, in run
step_func()
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/tools/subprocess_wrapper.py”, line 153, in parent_wrapper
func(self, *args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 374, in step1
self._update_fp_data()
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/flows/optimization_flow.py”, line 360, in _update_fp_data
self.set_fp_params(self._model.export_weights())
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 604, in export_weights
return self._export_npz(mode=NpzExportMode.WEIGHTS, include_shared_weights=include_shared_weights)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 613, in _export_npz
self.build(shapes)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/keras/src/layers/layer.py”, line 226, in build_wrapper
original_build_method(*args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/utils/distributed_utils.py”, line 122, in wrapper
res = func(self, *args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1260, in build
layer.build(layer.input_shapes)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/keras/src/layers/layer.py”, line 226, in build_wrapper
original_build_method(*args, **kwargs)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 1570, in build
self.verify_layer_inputs_shape(input_shape)
File “/home/yuri/Source/hailo/hailo/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_conv_add.py”, line 204, in verify_layer_inputs_shape
raise InvalidInputShape(
hailo_model_optimization.acceleras.utils.acceleras_exceptions.InvalidInputShape: Input shapes [[None, 120, 208, 16], [None, 120, 208, 1]] doesn’t match each other in octonet/conv2
```