Hailo Compiler Failed Recording Best-Effort Buffers

General
, ,
1

When compiling the quantized HAR model to HEF the following error is encountered, what could be the possible cause and solution?

command

hailo compiler --hw-arch hailo8l model_optimized.har

Error

[warning] Failed recording best-effort buffers: ddr_portal_from_dw23_defuse_reshape_hxf_to_w_transposed_to_dw23_defuse_1x1 was not found in toposort recipe during buffers override record
[error] Mapping Failed (Timeout, allocation time: 3h 20m 4s)
[error] Mapping Failed (Timeout, allocation time: 3h 20m 5s)
Resolver didn't find possible solution.
Watchdog expired after 1h 0m 0s
Mapping Failed (Timeout, allocation time: 3h 20m 5s)
[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: Resolver didn't find possible solution.
Watchdog expired after 1h 0m 0s
Mapping Failed (Timeout, allocation time: 3h 20m 5s)
2

Hey @m.wijkhuizen

The warning message:

[warning] Failed recording best-effort buffers: ddr_portal_from_dw23_defuse_reshape_hxf_to_w_transposed_to_dw23_defuse_1x1 was not found in toposort recipe during buffers override record

indicates that the compiler encountered an issue while managing or allocating specific buffers related to the operation ddr_portal_from_dw23_defuse_reshape_hxf_to_w_transposed_to_dw23_defuse_1x1. This operation was expected in the compiler’s topological sort (toposort) of model components but was not found, potentially affecting buffer allocation and model execution efficiency.

What This Warning Implies

  1. Unrecognized or Unsupported Operation:

    • This operation likely represents a reshaping or transposition within your model, and it might not be fully supported in this context. Such complex data manipulations can sometimes challenge the Hailo compiler’s optimization processes.

  2. Buffer Allocation Mismatch:

    • The compiler sequences model operations in a specific order (toposort) to allocate memory efficiently. If this operation doesn’t align with the expected order, it may lead to mismatches during buffer recording, impacting memory allocation.

Possible Causes and Solutions

  1. Simplify Model Operations:

    • Consider simplifying the model by reviewing where reshaping, transposing, or similar operations are used. Reducing or reconfiguring non-critical transformations can sometimes help the compiler recognize the model structure more effectively.

  2. Check for Operation Compatibility:

    • Verify that the operations used in the model are fully compatible with Hailo’s supported operations. If this particular operation is essential, try reformatting it in a more straightforward or supported way to facilitate the compiler’s processing.

  3. Optimize Compiler Settings:

    • The Hailo compiler provides optimization options (such as --performance flags) that prioritize allocation speed or memory efficiency. These may help the compiler handle the buffer allocation phase more effectively with models involving complex operations.

This warning suggests a potential misalignment in buffer management expectations. Addressing it by simplifying the model or optimizing settings can support a smoother compilation and enhance model performance on Hailo hardware.

3

Could you elaborate on point 3?
When running:

hailo compiler --help

I do not see any optimization options, where could I find a list with possible configurations?

4

To add some additional context, the model is quite large with ~40M parameters and to preserve as much precision as possible the quantization process is in 16 bits only.

The following line is logged, which indicates a maximum utilization of 60%? It is unclear why I would limit the utilization to 60%.

How would I set all these values to 100% and configure the compiler to not focus on speed but simply find a way to run the model.

[info] Resources optimization params: max_control_utilization=60%, max_compute_utilization=60%, max_compute_16bit_utilization=60%, max_memory_utilization (weights)=60%, max_input_aligner_utilization=60%, max_apu_utilization=60%