Hi, I am trying to introduce a reshape layer to my model: BCHW to BCP (where P=H*W) to have a model with convolutional backbone and visual transformer on top of it.
In reality dimensions I am interested in are B=1, C=256, H=W=96 (at least).
I created a simple script that would convert a torch model to ONNX format, simplify it and then translate to HAR, optimize, quantize and compile using hailo_sdk_client.
I noticed that it works ok for C=W=H=64 (example_0.py), but fails when I double either C,H or W (example_1.py sets C=128). It seemed like some memory limitation so I tried to split the tensor into two, perform the reshape and concatenate it (example_2.py), but it resulted in an error that I don’t understand.
According to the documentation Supported reshape: “Conv like” to “Dense like” Reshape there should be a way of performing this basic operation even on large tensors.
Could you help me with this task?
Here is the python script example_0.py
# example_0.py
def minimal_example():
import os
import tempfile
from pathlib import Path
import numpy as np
import onnx
import onnxsim
import torch
from hailo_sdk_client import ClientRunner
class SimpleBCHWToBCP(torch.nn.Module):
def forward(self, x: torch.Tensor):
B, C, H, W = x.shape
P = H * W
x = x.reshape(B, C, P)
return x
class SequentialBCHWToBCP(torch.nn.Module):
def forward(self, x: torch.Tensor):
B, C, H, W = x.shape
P = H * W
x_1, x_2 = torch.split(x, 64, dim=1)
x_1 = x_1.reshape(B, -1, P)
x_2 = x_2.reshape(B, -1, P)
x = torch.cat([x_1, x_2], dim=1)
return x
os.chdir(tempfile.mkdtemp(dir=Path(__file__).parent.resolve()))
ONNX_PATH = f"{__file__.removesuffix('.py')}.onnx"
OPSET_VERSION = 15
IN_SHAPE = (1, 64, 64, 64) # BCHW
torch_input = torch.randn(IN_SHAPE)
# model = SimpleBCHWToBCP()
model = SequentialBCHWToBCP()
# =============== Convert to .ONNX =================
model.eval()
with torch.no_grad():
torch.onnx.export(
model,
torch_input,
f=ONNX_PATH,
verbose=False,
do_constant_folding=False,
opset_version=OPSET_VERSION, # Use the appropriate ONNX opset version
input_names=["input"],
output_names=["output"],
)
onnx_model, _ = onnxsim.simplify(ONNX_PATH)
onnx.checker.check_model(onnx_model, full_check=True)
onnx.save_model(onnx_model, ONNX_PATH)
print(f"Saved ONNX model to {ONNX_PATH}")
# ============== Parse the model (.onnx -> .hef) ===============
print(f"{onnx.__version__ = }")
print(f"{torch.__version__ = }")
print(f"{__import__('hailo_sdk_client').__version__ = }")
runner = ClientRunner(hw_arch="hailo8l")
runner.translate_onnx_model(ONNX_PATH)
calib_data = np.random.randn(64, *IN_SHAPE[1:]).astype(np.float32)
calib_data = calib_data.transpose(0, 2, 3, 1) # BCHW -> BHWC - WHY ???
print(f"Optimizing model with {calib_data.shape = }")
runner.optimize(calib_data=calib_data)
print("Compiling model...")
runner.compile()
if __name__ == "__main__":
minimal_example()
with the corresponding diffs I described
(hailo_virtualenv) hailo@quczer-seagle:/mnt/bchw_to_bcp$ diff example_0.py example_1.py
1,2d0
< # example_0.py
<
38c36
< IN_SHAPE = (1, 64, 64, 64) # BCHW
---
> IN_SHAPE = (1, 128, 64, 64) # BCHW
(hailo_virtualenv) hailo@quczer-seagle:/mnt/bchw_to_bcp$ diff example_1.py example_2.py
39,40c39,40
< model = SimpleBCHWToBCP()
< # model = SequentialBCHWToBCP()
---
> # model = SimpleBCHWToBCP()
> model = SequentialBCHWToBCP()
I am working in Hailo AI Suite container.
Here are the fragments of errors I got:
(hailo_virtualenv) hailo@quczer-seagle:/mnt/bchw_to_bcp$ python example_1.py
Saved ONNX model to /mnt/bchw_to_bcp/example_1.onnx
onnx.__version__ = '1.16.0'
torch.__version__ = '2.6.0+cu124'
__import__('hailo_sdk_client').__version__ = '3.31.0'
[info] Translation started on ONNX model Model
...
[info] Translation completed on ONNX model model (completion time: 00:00:00.01)
Optimizing model with calib_data.shape = (64, 64, 64, 128)
[info] Starting Model Optimization
...
[info] Model Optimization is done
Compiling model...
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
...
[info] Successfully built optimization options - 0s 850ms
[error] Mapping Failed (allocation time: 0s)
No successful assignments: format_conversion1 errors:
Agent infeasible
format_conversion1 errors:
Agent infeasible
...
format_conversion1 errors:
Agent infeasible
[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No successful assignments: format_conversion1 errors:
Agent infeasible
format_conversion1 errors:
Agent infeasible
...
format_conversion1 errors:
Agent infeasible
and
(hailo_virtualenv) hailo@quczer-seagle:/mnt/bchw_to_bcp$ python example_2.py
Saved ONNX model to /mnt/bchw_to_bcp/example_2.onnx
onnx.__version__ = '1.16.0'
torch.__version__ = '2.6.0+cu124'
__import__('hailo_sdk_client').__version__ = '3.31.0'
[info] Translation started on ONNX model model
...
[info] Model Optimization is done
Compiling model...
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
Layer output_layer1 can't spatial reshape from (1, 4096, 128) to (64, 64, 128) without a successor format_conversion.
consider adding the following command to model script: "reshape_layer = format_conversion(output_layer1, concat1, hailo_rgb_to_tf_rgb)"
[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: Layer output_layer1 can't spatial reshape from (1, 4096, 128) to (64, 64, 128) without a successor format_conversion.
consider adding the following command to model script: "reshape_layer = format_conversion(output_layer1, concat1, hailo_rgb_to_tf_rgb)"
I also attach the ONNX representations for both models.
Thank you for your help!
Best regards,
Michał