Compilation failed with unexpected crash

When compiling a model to hef format, Im getting an error which is hard to interpret:
BackendAllocatorException: Compilation failed with unexpected crash
Parsing and Optimization steps were successful.

acceleras.log

2025-02-11 11:48:00,343 - INFO - mo_config.py:156 - Using default optimization level of 2
2025-02-11 11:48:00,357 - INFO - mo_script_parser.py:82 - Model received quantization params from the hn
2025-02-11 11:48:06,539 - INFO - algorithm_base.py:143 - Starting Mixed Precision
2025-02-11 11:48:07,146 - INFO - algorithm_base.py:159 - Mixed Precision is done (completion time is 00:00:00.60)
2025-02-11 11:48:08,350 - INFO - algorithm_base.py:140 - LayerNorm Decomposition skipped
2025-02-11 11:48:10,887 - INFO - algorithm_base.py:143 - Starting Statistics Collector
2025-02-11 11:48:10,901 - INFO - stats_collection.py:76 - Using dataset with 1024 entries for calibration
2025-02-11 11:50:27,807 - INFO - algorithm_base.py:159 - Statistics Collector is done (completion time is 00:02:16.90)
2025-02-11 11:50:27,911 - INFO - stats_collection.py:76 - Using dataset with 1024 entries for calibration
2025-02-11 11:52:57,968 - INFO - algorithm_base.py:143 - Starting Fix zp_comp Encoding
2025-02-11 11:52:57,970 - INFO - algorithm_base.py:159 - Fix zp_comp Encoding is done (completion time is 00:00:00.00)
2025-02-11 11:52:57,972 - INFO - algorithm_base.py:140 - Matmul Equalization skipped
2025-02-11 11:52:58,540 - INFO - avgpool_op.py:112 - No shifts available for layer model_converted/reducing_avgpool1/avgpool_op, using max shift instead. delta=2.0000
2025-02-11 11:53:01,399 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv33/conv_op, using max shift instead. delta=0.2848
2025-02-11 11:53:01,497 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv33/conv_op, using max shift instead. delta=0.1424
2025-02-11 11:53:01,737 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv36/conv_op, using max shift instead. delta=0.3458
2025-02-11 11:53:01,791 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv36/conv_op, using max shift instead. delta=0.1729
2025-02-11 11:53:01,880 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv33/conv_op, using max shift instead. delta=0.1424
2025-02-11 11:53:02,059 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv37/conv_op, using max shift instead. delta=1.0960
2025-02-11 11:53:02,108 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv37/conv_op, using max shift instead. delta=0.5480
2025-02-11 11:53:02,150 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv36/conv_op, using max shift instead. delta=0.1729
2025-02-11 11:53:02,190 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv38/conv_op, using max shift instead. delta=0.3861
2025-02-11 11:53:02,237 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv38/conv_op, using max shift instead. delta=0.1930
2025-02-11 11:53:02,277 - INFO - conv_stripped_op.py:523 - No shifts available for layer model_converted/conv37/conv_op, using max shift instead. delta=0.5480
2025-02-11 11:53:02,379 - INFO - activation_op.py:1817 - activation fitting started for model_converted/ew_sub_softmax1/act_op
2025-02-11 11:53:07,348 - INFO - activation_op.py:1817 - activation fitting started for model_converted/reduce_sum_softmax1/act_op
2025-02-11 11:54:00,558 - INFO - algorithm_base.py:140 - Finetune encoding skipped
2025-02-11 11:54:00,563 - INFO - algorithm_base.py:140 - Bias Correction skipped
2025-02-11 11:54:00,585 - INFO - algorithm_base.py:140 - Adaround skipped
2025-02-11 11:54:00,597 - INFO - algorithm_base.py:143 - Starting Quantization-Aware Fine-Tuning
2025-02-11 11:54:00,663 - INFO - qft.py:332 - Using dataset with 1024 entries for finetune
2025-02-11 12:17:52,943 - INFO - algorithm_base.py:159 - Quantization-Aware Fine-Tuning is done (completion time is 00:23:51.32)
2025-02-11 12:18:24,703 - INFO - algorithm_base.py:143 - Starting Layer Noise Analysis
2025-02-11 12:20:53,228 - INFO - algorithm_base.py:159 - Layer Noise Analysis is done (completion time is 00:02:28.52)
2025-02-11 12:20:54,159 - INFO - quant_checker.py:122 - Output layers signal-to-noise ratio (SNR): measures the quantization noise (higher is better)
2025-02-11 12:20:54,160 - INFO - quant_checker.py:127 - 	model_converted/output_layer1 SNR:	28.82 dB

client.log:

2025-02-11 11:44:08,384 - INFO - parser.py:158 - Translation started on ONNX model model_converted
2025-02-11 11:44:08,547 - INFO - parser.py:193 - Restored ONNX model model_converted (completion time: 00:00:00.16)
2025-02-11 11:44:09,388 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:01.00)
2025-02-11 11:44:10,712 - INFO - parser.py:373 - Start nodes mapped from original model: 'input': 'model_converted/input_layer1'.
2025-02-11 11:44:10,713 - INFO - parser.py:374 - End nodes mapped from original model: 'onnx_graphsurgeon_node_2'.
2025-02-11 11:44:10,770 - INFO - parser.py:288 - Translation completed on ONNX model model_converted (completion time: 00:00:02.39)
2025-02-11 11:44:11,861 - INFO - client_runner.py:1831 - Saved HAR to: /root/ai_kit/bringup/model/21_01_2025/model_converted_hailo_model.har
2025-02-11 11:44:16,033 - INFO - main.py:34 - Current Time: 11:44:16, 02/11/25
2025-02-11 11:44:17,089 - INFO - main.py:46 - CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) W-2275 CPU @ 3.30GHz, Number Of Cores: 28, Utilization: 0.6%
2025-02-11 11:44:17,089 - INFO - main.py:50 - Memory: Total: 62GB, Available: 51GB
2025-02-11 11:44:17,090 - INFO - main.py:54 - System info: OS: Linux, Kernel: 5.15.0-130-generic
2025-02-11 11:44:17,090 - INFO - main.py:55 - Hailo DFC Version: 3.29.0
2025-02-11 11:44:17,138 - INFO - main.py:60 - HailoRT Version: Not Installed
2025-02-11 11:44:17,142 - INFO - main.py:76 - PCIe: No Hailo PCIe device was found
2025-02-11 11:44:17,142 - INFO - main.py:109 - Running `hailo visualizer ./model/21_01_2025/model_converted_hailo_model.har --no-browser`
2025-02-11 11:47:58,584 - INFO - client_runner.py:479 - Loading model script commands to model_converted from string
2025-02-11 11:47:59,218 - IMPORTANT - sdk_backend.py:1009 - Starting Model Optimization
2025-02-11 12:21:07,437 - IMPORTANT - sdk_backend.py:1050 - Model Optimization is done
2025-02-11 12:21:09,043 - INFO - client_runner.py:1831 - Saved HAR to: /root/ai_kit/bringup/model/21_01_2025/model_converted_quantized_model.har
2025-02-11 12:21:14,168 - WARNING - model_script_parser.py:374 - Model script is empty
2025-02-11 12:21:14,169 - INFO - client_runner.py:479 - Loading model script commands to model_converted from string
2025-02-11 12:21:14,176 - WARNING - model_script_parser.py:374 - Model script is empty
2025-02-11 12:21:14,237 - INFO - hailo_tools_runner.py:507 - To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
2025-02-11 12:40:01,129 - ERROR - hailo_tools_runner.py:579 - Failed to produce compiled graph

final lines of core.log (too large to post full log):

[2025-02-11 12:32:58.781] [default] [info] No successful assignment for conv33_sd9_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for conv33_sd9_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for conv33_sd9_defuse_1x1, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for sh_from_sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_avgpool9_conv32_to_conv, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for sh_from_sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_avgpool9_conv32_to_co_1, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for concat_from_conv33_sd0_defuse_1x1_conv33_sd1_defuse_1x1_conv33_sd2_defuse_1x1_conv33_sd3_defuse_1x1_to_conv33_sdc, most common errors: 
[2025-02-11 12:32:58.781] [default] [info] No successful assignment for concat_from_conv33_sd4_defuse_1x1_conv33_sd5_defuse_1x1_conv33_sd6_defuse_1x1_conv33_sd7_defuse_1x1_to_conv33_sdc, most common errors: 
[2025-02-11 12:36:26.356] [default] [info] Iteration #32 - Contexts: 3, Changed: context_1, Fast FPS: 4.28353 (best: 0), Failed on: Automri finished with too many resources on context_1
[2025-02-11 12:36:26.356] [default] [info] Iteration #32 - Contexts: 3
[2025-02-11 12:36:26.983] [default] [info] Iteration #33 - Contexts: 3, Changed: context_1, Fast FPS: 4.28344 (best: 0), Failed on: Automri finished with too many resources on context_1
[2025-02-11 12:36:26.983] [default] [info] Iteration #33 - Contexts: 3
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd0_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd0_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: Agent infeasible
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd5_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for conv34_sd5_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd0_defuse_width_feature_reshape_conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.900] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: Agent infeasible
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.928] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: Agent infeasible
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd5_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd5_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd6_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd6_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd6_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd0_defuse_width_feature_reshape_conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: Agent infeasible
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd4_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:27.932] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd0_defuse_width_feature_reshape_conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.932] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd0_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd0_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd4_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd4_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd5_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd5_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd6_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd6_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd6_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd7_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd7_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd7_defuse_1x1, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for conv34_sd8_defuse_1x1, most common errors: Agent infeasible
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:27.995] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd0_defuse_width_feature_reshape_conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd4_defuse_width_feature_reshape_conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:27.995] [default] [info] No successful assignment for concat_from_conv34_sd4_defuse_1x1_conv34_sd5_defuse_1x1_conv34_sd6_defuse_1x1_conv34_sd7_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd0_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd0_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd4_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd4_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd5_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd5_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd6_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd6_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd6_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd7_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd7_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd7_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd8_defuse_1x1, most common errors: Agent infeasible
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd8_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd9_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd9_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for conv34_sd9_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd0_defuse_width_feature_reshape_conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd4_defuse_width_feature_reshape_conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:28.064] [default] [info] No successful assignment for concat_from_conv34_sd4_defuse_1x1_conv34_sd5_defuse_1x1_conv34_sd6_defuse_1x1_conv34_sd7_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for concat_from_conv32-34_to_concat1_dummy_37,111, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sdc, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd0_defuse_1x1, most common errors: Agent infeasible
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 1 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 2 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 3 subclusters
conv34_sd0_defuse_1x1: Minimum needed amount of subclusters was not reached. Needed: 5, but requests only 4 subclusters

[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd1_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd1_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd1_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd2_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd2_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd2_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd3_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd3_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd3_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd4_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd4_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd4_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd5_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd5_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd6_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd6_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd6_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd7_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd7_defuse_reshape_hxf_to_w_transposed, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for conv34_sd7_defuse_1x1, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for sh_from_sh_from_conv31_to_avgpool9_conv32-34_to_conv34_sd4_defuse_width_feature_reshape_conv34_sd5_defuse_width_feature_reshape, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for concat_from_conv34_sd0_defuse_1x1_conv34_sd1_defuse_1x1_conv34_sd2_defuse_1x1_conv34_sd3_defuse_1x1_to_conv34_sdc, most common errors: 
[2025-02-11 12:36:28.141] [default] [info] No successful assignment for concat_from_conv34_sd4_defuse_1x1_conv34_sd5_defuse_1x1_conv34_sd6_defuse_1x1_conv34_sd7_defuse_1x1_to_conv34_sdc, most common errors: 

When using the tutorials model (resnet_v1_18) the full process (parsing + optimizing + compiling) worked without issues. A possible hint is that the model I’m trying to compile is significantly larger than the one from the tutorial:

  • 109 vs 32 layers,
  • 93.3 vs 3.6 GOPS
  • 1024x1216x3 vs 224x224x3 input shape
  • 1024x1216x3 vs 1000 output shape

Another possible hint may be that some steps in the optimization were skipped, as can be seen in the logs (1024 images were used for the optimization):

  • Matmul Equalization
  • Finetune encoding
  • Bias Correction
  • Adaround

In the compilation step it stops at iteration 35 with a bunch of traced functions which couldn’t be processed (in the logs above):

Hey @Miguel_Silva ,

Welcome to the Hailo Community!

From the logs you provided i see the following issues :

  1. Model exceeds available hardware resources in terms of layers and computational load
  2. Quantization may degrade accuracy due to low bit representation
  3. Insufficient number of available subclusters for complex layers
  4. Advanced optimization steps (Finetune and Equalization) were skipped

Solutions

1. For Resource Constraints

# Adjust optimization level
model_optimization_flavor(optimization_level=2)

# For higher compression and performance
model_optimization_flavor(compression_level=1)

2. For Quantization Issues

# Enable 16-bit precision mode for critical layers
quantization_param(conv1, precision_mode="a16_w16")

3. For Subcluster Allocation

# Reduce required subclusters per layer
allocator_param(max_subclusters_per_layer=4)

4. For Missing Optimizations

# Enable fine-tuning steps
post_quantization_optimization(
    finetune,
    policy="enabled",
    epochs=8,
    learning_rate=0.0001
)

Each solution aims to address a specific compilation issue while optimizing model performance. Please try these fixes and update me with the results !

Hi @omria , thank you for the reply and suggestions!

Solution 3 doesn’t seem to be possible, since the tool does not recognize the “max_subclusters_per_layer” parameter for allocator_param. I could also not find it in the user guide (Release 3.29.0).
I looked into the guide for something similar and found this:
compilation_param({conv*}, resources_allocation_strategy=manual_scs_selection, number_of_subclusters=4)
However, setting this makes the compilation crash immediately (no iterations/context creation).

Regarding the other solutions, I tried them out, and there were some changes in the behavior. Now the compilation crashes at an earlier iteration (26th) and it uses 2 contexts instead of 3. The error is the same though:
compilation failed with unexpected crash