Failed during compilation with "map at"

General
1

Here are my logs the error message is pretty unhelpful. I am guessing its some memory error but not sure? I removed quite a bit of logs to get below post limit.

[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[info] Finding the best partition to contexts...
e[?25l[info] Iteration #1 - Contexts: 4
...
[info] Iteration #57 - Contexts: 4
e[?25h
[info] Using Multi-context flow
[info] Resources optimization guidelines: Strategy -> GREEDY Objective -> MAX_FPS
[info] Resources optimization params: max_control_utilization=60%, max_compute_utilization=60%, max_compute_16bit_utilization=60%, max_memory_utilization (weights)=60%, max_input_aligner_utilization=60%, max_apu_utilization=60%
[info] Solving the allocation (Mapping), time per context: 59m 59s

[info] context_0 (context_0):
Iterations: 4
Reverts on cluster mapping: 0
Reverts on inter-cluster connectivity: 0
Reverts on pre-mapping validation: 0
Reverts on split failed: 0
[info] context_1 (context_1):
Iterations: 4
Reverts on cluster mapping: 0
Reverts on inter-cluster connectivity: 0
Reverts on pre-mapping validation: 0
Reverts on split failed: 0
[info] context_2 (context_2):
Iterations: 4
Reverts on cluster mapping: 0
Reverts on inter-cluster connectivity: 0
Reverts on pre-mapping validation: 0
Reverts on split failed: 0
[info] context_3 (context_3):
Iterations: 4
Reverts on cluster mapping: 0
Reverts on inter-cluster connectivity: 0
Reverts on pre-mapping validation: 0
Reverts on split failed: 0
[info] context_0 utilization: 
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Cluster   | Control Utilization | Compute Utilization | Memory Utilization |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | cluster_0 | 100%                | 25%                 | 30.5%              |
[info] | cluster_1 | 43.8%               | 10.9%               | 14.8%              |
[info] | cluster_4 | 56.3%               | 14.1%               | 20.3%              |
[info] | cluster_5 | 56.3%               | 15.6%               | 32.8%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Total     | 64.1%               | 16.4%               | 24.6%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] context_1 utilization: 
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Cluster   | Control Utilization | Compute Utilization | Memory Utilization |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | cluster_0 | 50%                 | 15.6%               | 18.8%              |
[info] | cluster_1 | 75%                 | 20.3%               | 27.3%              |
[info] | cluster_4 | 100%                | 29.7%               | 30.5%              |
[info] | cluster_5 | 75%                 | 21.9%               | 31.3%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Total     | 75%                 | 21.9%               | 27%                |
[info] +-----------+---------------------+---------------------+--------------------+
[info] context_2 utilization: 
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Cluster   | Control Utilization | Compute Utilization | Memory Utilization |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | cluster_0 | 68.8%               | 17.2%               | 21.9%              |
[info] | cluster_1 | 50%                 | 12.5%               | 17.2%              |
[info] | cluster_4 | 100%                | 25%                 | 36.7%              |
[info] | cluster_5 | 37.5%               | 9.4%                | 14.8%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Total     | 64.1%               | 16%                 | 22.7%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] context_3 utilization: 
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Cluster   | Control Utilization | Compute Utilization | Memory Utilization |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | cluster_0 | 68.8%               | 17.2%               | 22.7%              |
[info] | cluster_1 | 6.3%                | 1.6%                | 2.3%               |
[info] | cluster_4 | 62.5%               | 18.8%               | 19.5%              |
[info] +-----------+---------------------+---------------------+--------------------+
[info] | Total     | 34.4%               | 9.4%                | 11.1%              |
[info] +-----------+---------------------+---------------------+--------------------+
[warning] Failed dump best-effort allocation proto: map::at
[info] Successful Mapping (allocation time: 1m 50s)
[info] Compiling context_0...
[info] Compiling context_1...
[info] Compiling context_2...
[info] Compiling context_3...
[info] Bandwidth of model inputs: 15.8203 Mbps, outputs: 47.4609 Mbps (for a single frame)
[info] Bandwidth of DDR buffers: 0.0 Mbps (for a single frame)
[info] Bandwidth of inter context tensors: 221.045 Mbps (for a single frame)
[error] Compilation failed with error: 'map::at'
e[?25h
[error] Failed to produce compiled graph
map::at
2

Any way to get more informative log on this, or how to debug this so I know how to change the model so it will compile?

3

Hi @smat,
On the running directory you should see a hailo.core.log, it’s a bit cryptic, but we know how to get stuff out of it.

4

What should I look for? I do see these logs now that you mention them.

5

Look from the bottom up for an error. You can DM me that file, and I’ll look as well.

6

Hi @Nadav I’m having the same issue and tried DM’ing you. There is no error in my core log.

7

Hi @l.hammond - can you share a few details on the model that you are compiling like how many parameters, is it a CNN or a transformer based, how many context were on the hef?

8

Hi @Nadav -
The model we’re currently trying to compile is a fine-tuned CNN-based YOLOv8n model from Ultralytics with 3.2M parameters. Input shape is (1,3,576,384) with a dtype of float32. According to hailo.core.log, the whole model fits in a single context

9

@Nadav now I’m not sure on the number of contexts. Where do I check to know for sure?

10

Hello @Nadav do you have any recommendations?

11

The model and it’s hyper parameters seems legite. Would you allow that I will try to compile that on my end?

12

@Nadav cool … what all would you need. the har?

13

onnx is better, so we can test with different end-nodes etc. Har can also work

14

I can send it to you. I just realized that this model was trained by a colleague on a yolo model that did not come from Hailo’s model zoo. Should that matter?

15

Hi @Nadav I DM’d you a google drive link