DAMO-YOLO compile errors

General
,
1

Hi there,

I’ve successfully a quantised a custom DAMO-YOLO model damoyolo_tinynasL25_S using the DFC running in a docker container. The model has good performance compared to the ONNX model from which it originated.

Here are the end node names I used:

 '/head/gfl_reg.0/Conv'
 '/head/gfl_reg.1/Conv'
'/head/gfl_reg.2/Conv'
'/head/gfl_cls.0/Conv'
'/head/gfl_cls.1/Conv'
'/head/gfl_cls.2/Conv'

Here is the ALLS config:

post_quantization_optimization(finetune, policy=disabled)
model_optimization_flavor(compression_level=1)
allocator_param(max_auto_merge_depth=3, merge_min_layer_utilization=0)
resources_param(max_utilization=0.975, max_compute_utilization=0.975, max_compute_16bit_utilization=0.975,
max_memory_utilization=0.90, max_input_aligner_utilization=0.975, max_apu_utilization=0.975)
damoyolo_tinynasL25_S = network_group([damoyolo_tinynasL25_S])

Note: this is from the Hailo model Zoo but I removed the ‘place’ commands as the layer names had changed and I couldn’t figure out how they mapped on to each other (could this be the issue?).

Now the ClientRunner seems to be getting stuck trying to compile the model:

[info] Context:0/0 Iteration 24: Trying parallel mapping...  
          cluster_0  cluster_1  cluster_2  cluster_3  cluster_4  cluster_5  cluster_6  cluster_7  prepost 
 worker0  *          *          *          *          *          *          *          *          V       
 worker1  *          *          *          *          *          *          *          *          V       
 worker2  *          *          *          *          *          *          *          *          V       
 worker3  *          *          *          *          *          *          *          *          V       

  16:11
[error] Preparing pasiflora, iteration: 27: Negative size is: 1
[error] Preparing pasiflora, iteration: 28: Negative size is: 1
[error] Preparing pasiflora, iteration: 26: Negative size is: 1
[error] Preparing pasiflora, iteration: 25: Negative size is: 1
[info] Context:0/0 Iteration 24: Trying parallel splits...   
          cluster_0  cluster_1  cluster_2  cluster_3  cluster_4  cluster_5  cluster_6  cluster_7  prepost

Any help on this would be greatly appreciated.

2

Hey @piers.turner ,

It looks like you getting the “Negative size is: 1” error, this issue typically occurs due to incorrect allocation or an incompatibility in layer merging or resource allocation.

Possible Causes & Solutions:

  1. Layer Placement Issues

    • Since you removed the ‘place’ commands, the DFC might be struggling to map the layers correctly
    • Try using DFC Studio to visualize and correctly assign layer placements
    • If the layer names changed, check the Hailo Parser logs to see the suggested start/end nodes

  2. Incorrect Resource Allocation

    • The error log shows parallel cluster allocation issues, which might be due to exceeding the allowed utilization
    • Modify resources_param values:
    resources_param(
    max_utilization=0.95, 
    max_compute_utilization=0.95, 
    max_compute_16bit_utilization=0.95,
    max_memory_utilization=0.85, 
    max_input_aligner_utilization=0.95, 
    max_apu_utilization=0.95
)
    • Lowering the utilization slightly (e.g., from 0.975 to 0.95) can prevent over-allocation failures