Error Issue occurred while processing Hailo 8 compiler

Hi Hailo Tech team, we are using Hailo Dataflow Compiler-3.27-Linux version and compiling our modal in order to deploy to Hailo 8 module. The modal was passed parser and quantization, after this, the quantized modal was around 8GB.
While when go to the compilation process, the error was shown as the follow error info. Please give your comments and suggestions on how to make it right, thanks.

[info] Loading network parameters
[info]Starting Hailo allocation and compilation flow
[error] Mapping Failed(allocation time:0s)
Titan defuse for node matmull has failed

Hi @u2185645, can you share the onnx/network?

Thank you @u2185645 for sharing a test block of your model.
A few notes on that-

  1. The Hailo operates on 4-dim tensors, where the batch dim is always 1, and being set dynamically. In your test block the dims changes to 3 in the middle:
    image
  2. With the above, ops like Gather, reshape can typically pose challenges to convert the network, these can sometime be addressed by usign ONNX-simplifier, that comes pre-installed with the DFC, but not always.
  3. Transpose operation can also pose challenge, especially when it involves the batch (0th) dimension.
    image
  4. Last point is related to the original ONNX that you’ve tried to convert. 8GB ONNX, means roughly a 2G parameter model. This currently not supported usecase, see this Dataflow Compiler (DFC) Availability! link

Hi Nadav,

Thanks for your checking and notes. After following your instruction, there are some remaining questions required to be checked again. Please kindly give suggestions as well.

1.1

Regarding the issue of 4-dim tensors, there is also a situation of 3-dim tensors in the vit_tiny.patch16_224_un.onnx section of the Hailo official website model zoo

vit_tiny_patch16_224_un.onnx graph
[Screenshot see my email , the pic can nit be uploaded successfully]

our test_block.onnx graph
[Screenshot see my email ]

1.2
Even after changing the batch dim of the entire onnx model to 1, it still reports the same error
[info] Loading network parameters
[info]Starting Hailo allocation and compilation flow
[error] Mapping Failed(allocation time:0s)
Titan defuse for node matmull has failed

2.1

The test_block models provided have all passed DFC’s translate’onnx_model(), indicating that the model conversion is not a problem and there is no need to simplify onnx. In addition, there is a very similar structure in the model zoo on the official website, vit_tiny_patch16_224_un.onnx. The operator usage is the same, only the shape of the input and output is different.

In addition, the answer did not mention Matmul. Please explain the error message.

[info] Loading network parameters
[info]Starting Hailo allocation and compilation flow
[error] Mapping Failed(allocation time:0s)
Titan defuse for node matmull has failed

With above info, screenshots please see from my email. Could you help reply following questions:

  1. Regarding the matmul operator, does it not support features (channels) above 3? So why can the matmul in our test_block. onnx pass through runner. Translate_onnx_model() and runner. optimize()?
  2. If matmul does not support this writing method, what is the way to rewrite it?

Thanks very much.

Fei