Can we control whether operator fusion is applied or not in the Hailo dataflow compiler?

General
1

Can we control whether operator fusion is applied or not in the Hailo dataflow compiler?

For instance, is it possible to selectively apply operator fusion (e.g., conv+bn) in some cases and disable it in others as desired?

Also, can we choose which stage of the intermediate representation (IR) to work with?

Thank you.

2

Hey @GGURIg ,

Controlling Operator Fusion:

The Hailo Dataflow Compiler automatically applies operator fusion optimizations (e.g., conv+bn) during the compilation process to maximize performance and efficiency. Currently, fine-grained control to selectively enable or disable operator fusion for specific cases is not directly supported in the compiler. The optimization process is designed to ensure the best overall performance for the model.

If you need to bypass operator fusion, you can modify the ONNX model before compilation:

  • Separate fused operations into standalone layers in your ONNX export or use an ONNX graph editor to split fused nodes.
  • Use the ONNX simplifier cautiously to avoid re-fusing layers during preprocessing.

Working with Intermediate Representation (IR) Stages:

The Hailo Dataflow Compiler does not provide direct access to modify specific stages of the IR. However, you can use profiling and debugging tools such as hailortcli parse-hef and hailortcli profile to inspect the compiler’s outputs and understand intermediate representations indirectly.

If you require access to intermediate computations for advanced use cases, consider restructuring the ONNX model to include auxiliary outputs at desired stages.

Suggestions for Custom Scenarios:

  1. Preprocessing the Model:

    • Adjust the model’s structure before compilation to achieve the desired level of granularity for fused operations.

  2. Custom Pipelines:

    • For cases requiring specific IR manipulations, consider working with Hailo support for tailored solutions.

If operator fusion control or IR stage customization becomes available in future releases, we will provide detailed documentation and guidance.

Please let us know if you have any further questions or need assistance with model restructuring. We’re here to help!

Best regards,
Omria