ONNX to HEF Conversion - Mapping Failed

General
, ,
1

Hi,

I’m trying to convert a small model from ONNX to HEF to run on hailo8.
This model has 1 input (float32[ 1, 1024, 57, 150] ) and 1 output (float32[ 1, 19, 57, 450]).

This is the graph of the ONNX model:

image
image992×1264 70.6 KB

All phases of conversion from ONNX to HAR run successfully with the optimization running with the following script:

alls_model_script = [
    "model_optimization_flavor(optimization_level=2, compression_level=0, batch_size=1)\n",
    "performance_param(compiler_optimization_level=max)\n",
]

It is in the final Compilation step that I got the error:

[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 20/23 failures.

Mapping Failed (Timeout, allocation time: 31m 33s)

By analysing the tracelog it seems the compilation fails earlier with a “Failed to resolve prepost” message.
I can’t figure out what this means, as the model is a really simple network and no pre-processing is done (no normalization layers, etc).

I leave here the links to the ONNX file and the HAR file after optimization:
ONNX File
HAR Quantized Model

The compilation command I’m executing and that is failing is just:

runner = ClientRunner(har=quant_model_har_path)
hef = runner.compile()
with open(HEF_model_filename, "wb") as f:
    f.write(hef)

Here is the Full tracelog of the Compilation phase:

Tracelog 
[2024-08-29 17:21:07.666] [default] [info] Ran from command: "convert_SimpleModelToHailo.py"
[2024-08-29 17:21:07.667] [default] [info] Loading network parameters
[2024-08-29 17:21:07.702] [default] [info] Starting Hailo allocation and compilation flow
[2024-08-29 17:21:07.728] [default] [info] Model name: featcomb
[2024-08-29 17:22:43.871] [default] [info] Trying to solve in single context
[2024-08-29 17:22:46.380] [default] [info] Model fits in single context
[2024-08-29 17:22:46.380] [default] [info] Model fits in single context, trying to increase utilization
[2024-08-29 17:22:46.396] [default] [info] Initial resources counter with: SCs: 43/512, 16bit SCs: 0/512, APUs: 35/160, IAs: 35/256, L3 weights: 136/922, L3 ios: 168/922, LCUs: 35/154
[2024-08-29 17:24:36.071] [default] [info] Converged after 3 iterations
[2024-08-29 17:24:36.071] [default] [info] Final resources: SCs: 45/512, 16bit SCs: 0/512, APUs: 35/160, IAs: 35/256, L3 weights: 136/922, L3 ios: 42/922, LCUs: 35/154
[2024-08-29 17:24:36.071] [default] [info] Trying to apply higher utilization solution
[2024-08-29 17:24:38.969] [default] [info] Running Auto-Merger
[2024-08-29 17:24:41.613] [default] [info] Auto-Merger is done
[2024-08-29 17:25:04.270] [default] [info] Higher utilization solution failed validation: Failed to resolve prepost, skipping to the next one
[2024-08-29 17:25:04.270] [default] [info] Trying to apply higher utilization solution
[2024-08-29 17:25:06.822] [default] [info] Running Auto-Merger
[2024-08-29 17:25:09.317] [default] [info] Auto-Merger is done
[2024-08-29 17:25:28.034] [default] [info] Higher utilization solution failed validation: Failed to resolve prepost, skipping to the next one
[2024-08-29 17:25:31.046] [default] [info] Running Auto-Merger
[2024-08-29 17:25:33.628] [default] [info] Auto-Merger is done
[2024-08-29 17:26:05.595] [default] [info] Single context solution failed, skipping it
[2024-08-29 17:26:05.595] [default] [info] Single context flow failed: Failed to resolve prepost, continuing in multi context
[2024-08-29 17:27:33.101] [default] [info] Finding the best partition to contexts...
[2024-08-29 17:28:43.038] [default] [info] Running Auto-Merger
[2024-08-29 17:28:45.780] [default] [info] Auto-Merger is done
[2024-08-29 17:29:11.254] [default] [info] Iteration #1 - Contexts: 1, Changed: context_0, Fast FPS: 13.3787 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:29:34.075] [default] [info] Iteration #2 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:29:56.956] [default] [info] Iteration #3 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:30:57.728] [default] [info] Iteration #4 - Contexts: 2, Changed: context_0, Fast FPS: 13.324 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:32:19.194] [default] [info] Iteration #5 - Contexts: 2, Changed: context_0, Fast FPS: 13.3241 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:33:19.824] [default] [info] Iteration #6 - Contexts: 2, Changed: context_0, Fast FPS: 13.3241 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:34:21.454] [default] [info] Iteration #7 - Contexts: 2, Changed: context_0, Fast FPS: 13.324 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:35:00.261] [default] [info] Iteration #8 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:36:26.499] [default] [info] Iteration #9 - Contexts: 2, Changed: context_0, Fast FPS: 13.3241 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:40:24.945] [default] [info] Iteration #10 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:42:02.993] [default] [info] Iteration #11 - Contexts: 2, Changed: context_0, Fast FPS: 13.3237 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:42:41.722] [default] [info] Iteration #12 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:43:20.915] [default] [info] Iteration #13 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:44:00.351] [default] [info] Iteration #14 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:44:38.892] [default] [info] Iteration #15 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:47:00.789] [default] [info] Iteration #16 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:47:39.727] [default] [info] Iteration #17 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:48:18.311] [default] [info] Iteration #18 - Contexts: 2, Changed: context_0, Fast FPS: 13.3242 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:53:20.292] [default] [info] Iteration #19 - Contexts: 2, Changed: context_0, Fast FPS: 11.1377 (best: 0), Failed on: Validator failed on node: conv1_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d2 with Agent infeasible

[2024-08-29 17:53:49.729] [default] [info] Iteration #20 - Contexts: 2, Changed: context_0, Fast FPS: 11.0986 (best: 0), Failed on: Validator failed on node: conv1_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d2 with Agent infeasible

[2024-08-29 17:58:27.286] [default] [info] Iteration #21 - Contexts: 2, Changed: context_0, Fast FPS: 7.46989 (best: 0), Failed on: Validator failed on node: conv1_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d1 with Agent infeasible

[2024-08-29 17:58:41.254] [default] [info] Iteration #22 - Contexts: 2, Changed: context_0, Fast FPS: 12.4548 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:59:04.679] [default] [info] Iteration #23 - Contexts: 2, Changed: context_0, Fast FPS: 12.4508 (best: 0), Failed on: Failed to resolve prepost
[2024-08-29 17:59:04.679] [default] [info] Slowest context (context_0) has 1 layer (input_layer1)
[2024-08-29 17:59:06.451] [default] [info] e[1;41mMapping Failed (Timeout, allocation time: 31m 33s)e[0m
[2024-08-29 17:59:06.451] [default] [info] Builder fail with status 94: Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 20/23 failures.
Mapping Failed (Timeout, allocation time: 31m 33s)
[2024-08-29 17:59:06.451] [default] [info] Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 20/23 failures.
Mapping Failed (Timeout, allocation time: 31m 33s)

I’m using the latest Hailo AI Suite docker with DFC 3.28.

Can someone help or give any recommendation to compile this model ?

Thank you,

2

Hi,

I small follow-up on this problem.
I managed to make a small progress on this matter.

I started to remove the “split” and “slice” in the input and the “concat” in the output to simplify the model:

image
image1360×656 51.9 KB

The Compilation still failed with an (even more) cryptic message:

Compiler could not find a valid partition to contexts. Most commom error is: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible with 10/22 failures.

Full Tracelog 
[2024-08-30 11:30:52.703] [default] [info] Ran from command: "convert_featCombToHailo.py"
[2024-08-30 11:30:52.703] [default] [info] Loading network parameters
[2024-08-30 11:30:52.750] [default] [info] Starting Hailo allocation and compilation flow
[2024-08-30 11:30:52.773] [default] [info] Model name: famnet_featcomb
[2024-08-30 11:32:55.946] [default] [info] Trying to solve in single context
[2024-08-30 11:32:59.759] [default] [info] Model fits in single context
[2024-08-30 11:32:59.759] [default] [info] Model fits in single context, trying to increase utilization
[2024-08-30 11:32:59.779] [default] [info] Initial resources counter with: SCs: 38/384, 16bit SCs: 0/384, APUs: 28/120, IAs: 28/192, L3 weights: 173/768, L3 ios: 192/768, LCUs: 28/96
[2024-08-30 11:34:40.086] [default] [info] Converged after 4 iterations
[2024-08-30 11:34:40.086] [default] [info] Final resources: SCs: 41/384, 16bit SCs: 0/384, APUs: 28/120, IAs: 28/192, L3 weights: 173/768, L3 ios: 103/768, LCUs: 28/96
[2024-08-30 11:34:40.086] [default] [info] Trying to apply higher utilization solution
[2024-08-30 11:35:39.621] [default] [info] Higher utilization solution failed validation: Failed to find solution for large buffering << (24 L3s) between famnet_featcomb_scope2/conv3_defuse_reshape_hxf_to_w_transposed->famnet_featcomb_scope2/conv3_defuse_1x1_d0, famnet_featcomb_scope2/conv3_defuse_1x1_d1, skipping to the next one
[2024-08-30 11:35:39.621] [default] [info] Trying to apply higher utilization solution
[2024-08-30 11:36:10.496] [default] [info] Higher utilization solution failed validation: Failed to resolve prepost, skipping to the next one
[2024-08-30 11:37:16.226] [default] [info] Single context flow failed: Failed to find solution for large buffering << (54 L3s) between famnet_featcomb_scope2/conv3_defuse_reshape_hxf_to_w_transposed->famnet_featcomb_scope2/conv3_defuse_1x1_d0, famnet_featcomb_scope2/conv3_defuse_1x1_d1, continuing in multi context
[2024-08-30 11:39:10.955] [default] [info] Multi network model was compiled to multi context. This use case is not formally supported.
[2024-08-30 11:39:10.955] [default] [info] Finding the best partition to contexts...
[2024-08-30 11:41:42.537] [default] [info] Iteration #1 - Contexts: 1, Changed: context_0, Fast FPS: 13.4112 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv5_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv5_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:46:10.327] [default] [info] Iteration #2 - Contexts: 2, Changed: context_0, Fast FPS: 7.47999 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:50:27.838] [default] [info] Iteration #3 - Contexts: 2, Changed: context_0, Fast FPS: 7.45492 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 11:51:05.869] [default] [info] Iteration #4 - Contexts: 2, Changed: context_0, Fast FPS: 7.45492 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 11:51:43.912] [default] [info] Iteration #5 - Contexts: 2, Changed: context_0, Fast FPS: 7.45492 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 11:52:21.828] [default] [info] Iteration #6 - Contexts: 2, Changed: context_0, Fast FPS: 7.45515 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 11:56:46.128] [default] [info] Iteration #7 - Contexts: 2, Changed: context_0, Fast FPS: 11.105 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:57:09.861] [default] [info] Iteration #8 - Contexts: 2, Changed: context_0, Fast FPS: 11.0863 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:57:33.300] [default] [info] Iteration #9 - Contexts: 2, Changed: context_0, Fast FPS: 11.0863 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:57:56.697] [default] [info] Iteration #10 - Contexts: 2, Changed: context_0, Fast FPS: 11.0863 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 11:58:20.008] [default] [info] Iteration #11 - Contexts: 2, Changed: context_0, Fast FPS: 11.0865 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:01:32.115] [default] [info] Iteration #12 - Contexts: 2, Changed: context_0, Fast FPS: 13.0552 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv4_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv5_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv5_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:02:16.512] [default] [info] Iteration #13 - Contexts: 3, Changed: context_1, Fast FPS: 7.36796 (best: 0), Failed on: Failed to resolve prepost
[2024-08-30 12:02:52.771] [default] [info] Iteration #14 - Contexts: 3, Changed: context_1, Fast FPS: 7.34362 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 12:03:28.839] [default] [info] Iteration #15 - Contexts: 3, Changed: context_1, Fast FPS: 7.34385 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 12:04:04.884] [default] [info] Iteration #16 - Contexts: 3, Changed: context_1, Fast FPS: 7.34385 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 12:04:41.068] [default] [info] Iteration #17 - Contexts: 3, Changed: context_1, Fast FPS: 7.34385 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d1 with Agent infeasible
Validator failed on node: famnet_featcomb_scope2/conv3_defuse_1x1_d2 with Agent infeasible

[2024-08-30 12:05:13.262] [default] [info] Iteration #18 - Contexts: 3, Changed: context_1, Fast FPS: 10.9137 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:05:34.952] [default] [info] Iteration #19 - Contexts: 3, Changed: context_1, Fast FPS: 10.9136 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:05:56.702] [default] [info] Iteration #20 - Contexts: 3, Changed: context_1, Fast FPS: 10.9136 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:06:18.722] [default] [info] Iteration #21 - Contexts: 3, Changed: context_1, Fast FPS: 10.9136 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:06:40.303] [default] [info] Iteration #22 - Contexts: 3, Changed: context_1, Fast FPS: 10.9137 (best: 0), Failed on: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:06:40.303] [default] [info] Slowest context (context_1) has 1 layer (famnet_featcomb_scope1/conv2)
[2024-08-30 12:06:41.792] [default] [info] e[1;41mMapping Failed (allocation time: 33m 45s)e[0m
[2024-08-30 12:06:41.793] [default] [info] Builder fail with status 94: Compiler could not find a valid partition to contexts. Most commom error is: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible
 with 10/22 failures.
[2024-08-30 12:06:41.793] [default] [info] Compiler could not find a valid partition to contexts. Most commom error is: Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: famnet_featcomb_scope1/conv2_defuse_1x1_d1 with Agent infeasible
 with 10/22 failures.

Then, I split the model in two parts and tried to compile each part to see if the error occurred and where it was located.

For my surprise this part Compiled Successfully :

image
image690×638 32.2 KB

However, the other part of the model Failed :

image
image924×808 44 KB

with the original error message:

Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 2/4 failures.
Mapping Failed (Timeout, allocation time: 10m 40s)

Full Tracelog 
[2024-08-30 12:18:54.556] [default] [info] Ran from command: "convert_featCombToHailo.py"
[2024-08-30 12:18:54.556] [default] [info] Loading network parameters
[2024-08-30 12:18:54.582] [default] [info] Starting Hailo allocation and compilation flow
[2024-08-30 12:18:54.605] [default] [info] Model name: famnet_featcomb
[2024-08-30 12:20:18.701] [default] [info] Trying to solve in single context
[2024-08-30 12:20:22.068] [default] [info] Model fits in single context
[2024-08-30 12:20:22.068] [default] [info] Model fits in single context, trying to increase utilization
[2024-08-30 12:20:22.083] [default] [info] Initial resources counter with: SCs: 22/512, 16bit SCs: 0/512, APUs: 16/160, IAs: 16/256, L3 weights: 116/922, L3 ios: 142/922, LCUs: 16/154
[2024-08-30 12:21:59.216] [default] [info] Converged after 4 iterations
[2024-08-30 12:21:59.216] [default] [info] Final resources: SCs: 25/512, 16bit SCs: 0/512, APUs: 16/160, IAs: 16/256, L3 weights: 116/922, L3 ios: 21/922, LCUs: 16/154
[2024-08-30 12:21:59.216] [default] [info] Trying to apply higher utilization solution
[2024-08-30 12:22:03.330] [default] [info] Running Auto-Merger
[2024-08-30 12:22:04.049] [default] [info] Auto-Merger is done
[2024-08-30 12:22:36.775] [default] [info] Higher utilization solution failed validation: Failed to resolve prepost, skipping to the next one
[2024-08-30 12:22:36.775] [default] [info] Trying to apply higher utilization solution
[2024-08-30 12:22:40.381] [default] [info] Running Auto-Merger
[2024-08-30 12:22:40.976] [default] [info] Auto-Merger is done
[2024-08-30 12:23:04.517] [default] [info] Higher utilization solution failed validation: Failed to resolve prepost, skipping to the next one
[2024-08-30 12:23:08.493] [default] [info] Running Auto-Merger
[2024-08-30 12:23:09.178] [default] [info] Auto-Merger is done
[2024-08-30 12:23:41.107] [default] [info] Single context solution failed, skipping it
[2024-08-30 12:23:41.107] [default] [info] Single context flow failed: Failed to resolve prepost, continuing in multi context
[2024-08-30 12:24:55.428] [default] [info] Finding the best partition to contexts...
[2024-08-30 12:26:08.247] [default] [info] Running Auto-Merger
[2024-08-30 12:26:09.010] [default] [info] Auto-Merger is done
[2024-08-30 12:26:43.581] [default] [info] Iteration #1 - Contexts: 1, Changed: context_0, Fast FPS: 13.4222 (best: 0), Failed on: Failed to resolve prepost
[2024-08-30 12:30:57.271] [default] [info] Iteration #2 - Contexts: 2, Changed: context_0, Fast FPS: 7.48341 (best: 0), Failed on: Validator failed on node: conv1_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d2 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv2_defuse_1x1_d1 with Agent infeasible

[2024-08-30 12:35:04.729] [default] [info] Iteration #3 - Contexts: 2, Changed: context_0, Fast FPS: 7.48342 (best: 0), Failed on: Validator failed on node: conv1_defuse_1x1_d0 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d1 with Agent infeasible
Validator failed on node: conv1_defuse_1x1_d2 with Agent infeasible

[2024-08-30 12:35:34.896] [default] [info] Iteration #4 - Contexts: 2, Changed: context_0, Fast FPS: 12.6269 (best: 0), Failed on: Failed to resolve prepost
[2024-08-30 12:35:34.897] [default] [info] Slowest context (context_0) has 1 layer (input_layer1)
[2024-08-30 12:35:35.894] [default] [info] e[1;41mMapping Failed (Timeout, allocation time: 10m 40s)e[0m
[2024-08-30 12:35:35.896] [default] [info] Builder fail with status 94: Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 2/4 failures.
Mapping Failed (Timeout, allocation time: 10m 40s)
[2024-08-30 12:35:35.896] [default] [info] Compiler could not find a valid partition to contexts. Most commom error is: Failed to resolve prepost with 2/4 failures.
Mapping Failed (Timeout, allocation time: 10m 40s)

But this meant that the problem was located somewhere in this part of the model.

So, after some more digging, I found out that I can Compile this part of the model, IF I change the input tensor “feat3” to shape [1, 256, 57, 75] instead of the original [1, 512, 57, 75]. It takes a really long time, but it compiles.

However, unfortunately, I can’t use this input shape in my model…

I’m starting to think that this error may have something to do with some kind of resources exhaustion on the compilation phase. I’m using a 32GB RAM PC and I monitored the (global) RAM usage and it never peaked outside of the 24/25 GB zone…

I’ll continue to find out if I can compile the original model and avoid the error…

3

Hi @pedrosantos,

Could you please send me the onnx file by dm?