Inquiry about using negative values in the calibration dataset and potential workarounds

General Discussion
,
1

Hello Hailo Community,

I am currently working on quantizing a custom model (e.g., ResNet / MobileNet) and performing calibration. During this process, I encountered an issue regarding the input range of the calibration dataset.

AccelerasUnsupportedError: linear_model_scope1/matmul2/matmul_op: Matmul without transpose input and with negative range is not supported

My original model was trained with normalized inputs that include negative values (e.g., [-1, 1] or standard normalization with mean/std). However, it seems that using negative values directly in the calibration dataset causes issues [or is not officially recommended/supported].

I have two questions regarding this:

  1. Hardware/Software constraints: Could you explain the exact reason why the calibration dataset struggles with or restricts negative input values? Is it directly related to the NPU’s physical input layer strictly expecting uint8 [0, 255]?

  2. Recommended Workaround: What is the best practice to bypass this limitation? Should I feed unnormalized [0, 255] images as the calibration dataset and handle the negative shift internally using the normalization1 parameters (mean/std) in the Hailo Dataflow Compiler (DFC) .alls script?

Any detailed explanations or references to the official documentation would be highly appreciated.

Thank you in advance!

2

Hi @Changhyeon_Lee,

1. Why this happens:

The Hailo NPU’s MatMul implementation (without transposed input) only supports unsigned (non-negative) activation ranges. When the quantizer detects negative values flowing into that MatMul layer, it raises this error. This is a hardware constraint on how fixed-point multiplication is wired for data-driven MatMul ops.

2. Recommended fix:

Feed raw [0, 255] images as your calibration dataset and use the normalization command in your .alls script to handle the mean/std shift on-device:

normalization1 = normalization([123.675, 116.28, 103.53], [58.395, 57.12, 57.375])

Adjust the values to match your model’s training preprocessing. The DFC User Guide states: “If a normalization layer has been added to the model, the calibration set should not be normalized.”

At inference time, you also feed raw [0, 255] images - the on-device normalization layer handles the rest.

Thanks,
Michael.

1 Like
3

Thank you for the explanation regarding the MatMul hardware constraints.

Based on discussions in other forums, I came across information suggesting that negative values can be handled when using a transposed input, similar to how the MHA (Multi-Head Attention) structure operates. However, I have been unable to find any official documentation confirming this. Could you point me to the specific documents or sections where this behavior is explained?

Additionally, I would like to ask the following questions:

  1. Mechanism: How exactly does the Hailo hardware handle negative activations when the input is transposed?

  2. Custom Layer Implementation: If I want to build a custom layer that replicates this behavior, how should I structure it so that the Hailo Quantizer recognizes and processes it in the same manner as an MHA structure?

Looking forward to your guidance.