No valid partition found during compile

I’m trying to compile a custom trained yolov8s_pose model. I’ve gotten all the way to the last compile step and cannot get past it. The model structure is standard as far as I can tell so I’m not sure why it would be so complicated.

Here is the output:

hailo compiler bed_optimized.har --hw-arch hailo8 --model-script mscript.alls 

[info] Current Time: 09:13:19, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.1%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler bed_optimized.har --hw-arch hailo8 --model-script mscript.alls`
[info] Loading model script commands to bed from mscript.alls
[info] ParsedPerformanceParam command, setting optimization_level(max=2)
[info] Compiling network
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 11s)
No valid partition found

[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No valid partition found

And the model script I’m using is the one from hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s_pose.alls but with the compiler_optimisation_level added:

normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
change_output_activation(conv71, sigmoid)
change_output_activation(conv58, sigmoid)
change_output_activation(conv44, sigmoid)
pre_quantization_optimization(equalization, policy=disabled)
quantization_param(output_layer3, precision_mode=a16_w16)
quantization_param(output_layer6, precision_mode=a16_w16)
quantization_param(output_layer9, precision_mode=a16_w16)
post_quantization_optimization(finetune, policy=enabled, learning_rate=0.00015)
performance_param(compiler_optimization_level=max)

If I don’t add the script i get:

hailo compiler bed_optimized.har --hw-arch hailo8

[info] Current Time: 09:22:48, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.1%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler bed_optimized.har --hw-arch hailo8`
[info] Compiling network
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 11s)
No successful assignments: concat18 errors:
	Agent infeasible


[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No successful assignments: concat18 errors:
	Agent infeasible

I’m completely at a loss. This is my first time working with vision models so if you respond then please assume I know nothing.

Hi @rosslote,
the below line in your alls, tells the compiler to “push” as hard as it can the optimizations on throughput. Please try to remove that line, in many cases the more lenient context gives the compiler a path to success.

performance_param(compiler_optimization_level=max)

That’s just the same as the model zoo one:

hailo compiler bed_optimized.har --hw-arch hailo8 --model-script hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s_pose.alls 

[info] Current Time: 12:13:59, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.2%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler bed_optimized.har --hw-arch hailo8 --model-script hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s_pose.alls`
[info] Loading model script commands to bed from hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s_pose.alls
[info] Compiling network
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 12s)
No successful assignments: concat18 errors:
	Agent infeasible


[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No successful assignments: concat18 errors:
	Agent infeasible

You still seem to have this line:
performance_param(compiler_optimization_level=max)

Please try without it.

Where are you seeing that? The script I used has this:

normalization1 = normalization([0.0, 0.0, 0.0], [255.0, 255.0, 255.0])
change_output_activation(conv71, sigmoid)
change_output_activation(conv58, sigmoid)
change_output_activation(conv44, sigmoid)
pre_quantization_optimization(equalization, policy=disabled)
quantization_param(output_layer3, precision_mode=a16_w16)
quantization_param(output_layer6, precision_mode=a16_w16)
quantization_param(output_layer9, precision_mode=a16_w16)
post_quantization_optimization(finetune, policy=enabled, learning_rate=0.00015)

It even says in the output that I should try setting compiler_optimization_level to “max”

Okay. so I decided to just try a non custom yolov8s_pose.onnx model for my own sanity. I downloaded this from the model_zoo github page: https://hailo-model-zoo.s3.eu-west-2.amazonaws.com/PoseEstimation/yolov8/yolov8s/pretrained/2023-06-11/yolov8s_pose.zip

Here is the complete output of my attempt:

(hailo_virtualenv) hailo@ross1:/local/workspace$ hailo parser onnx /local/shared_with_docker/yolov8s_pose.onnx 
[info] Current Time: 13:12:45, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.1%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo parser onnx /local/shared_with_docker/yolov8s_pose.onnx`
[info] Translation started on ONNX model yolov8s_pose
[info] Restored ONNX model yolov8s_pose (completion time: 00:00:00.21)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.82)
[info] Simplified ONNX model for a parsing retry attempt (completion time: 00:00:01.95)
Parsing failed with recommendations for end node names: ['/model.22/Concat_4', '/model.22/Sigmoid_1', '/model.22/Mul_3'].
Would you like to parse again with the recommendation? (y/n) 
y
[info] According to recommendations, retrying parsing with end node names: ['/model.22/Concat_4', '/model.22/Sigmoid_1', '/model.22/Mul_3'].
[info] Translation started on ONNX model yolov8s_pose
[info] Restored ONNX model yolov8s_pose (completion time: 00:00:00.15)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.76)
[info] Start nodes mapped from original model: 'images': 'yolov8s_pose/input_layer1'.
[info] End nodes mapped from original model: '/model.22/Concat_4', '/model.22/Sigmoid_1', '/model.22/Mul_3'.
[info] Translation completed on ONNX model yolov8s_pose (completion time: 00:00:01.52)
[warning] hw_arch parameter not given, using the default hw_arch hailo8.
If another device is the target, please run again using one of hailo8, hailo8r, hailo8l, hailo15h, hailo15m, hailo15l, hailo10h
[info] Saved HAR to: /local/workspace/yolov8s_pose.har
(hailo_virtualenv) hailo@ross1:/local/workspace$ hailo optimize yolov8s_pose.har --calib-set-path calib_set.npy 
[info] Current Time: 13:13:58, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.1%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo optimize yolov8s_pose.har --calib-set-path calib_set.npy`
[info] Starting Model Optimization
[warning] Reducing optimization level to 0 (the accuracy won't be optimized and compression won't be used) because there's no available GPU
[warning] Running model optimization with zero level of optimization is not recommended for production use and might lead to suboptimal accuracy results
[info] Model received quantization params from the hn
[info] MatmulDecompose skipped
[info] Starting Mixed Precision
[info] Model Optimization Algorithm Mixed Precision is done (completion time is 00:00:00.78)
[info] LayerNorm Decomposition skipped
[info] Starting Statistics Collector
[info] Using dataset with 64 entries for calibration
Calibration: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:47<00:00,  1.34entries/s]
[info] Model Optimization Algorithm Statistics Collector is done (completion time is 00:00:50.24)
[info] Output layer yolov8s_pose/activation1 with sigmoid activation was detected. Forcing its output range to be [0, 1] (original range was [1.0335516265058686e-07, 0.9999998807907104]).
[info] Starting Fix zp_comp Encoding
[info] Model Optimization Algorithm Fix zp_comp Encoding is done (completion time is 00:00:00.00)
[info] Matmul Equalization skipped
[info] Finetune encoding skipped
[info] Bias Correction skipped
[info] Adaround skipped
[info] Quantization-Aware Fine-Tuning skipped
[info] Layer Noise Analysis skipped
[info] Model Optimization is done
[info] Saved HAR to: /local/workspace/yolov8s_pose_optimized.har
(hailo_virtualenv) hailo@ross1:/local/workspace$ hailo compile -h
[info] Current Time: 13:15:59, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.2%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compile -h`
usage: hailo [-h] [--version]
             {fw-update,ssb-update,fw-config,udp-rate-limiter,fw-control,fw-logger,scan,sensor-config,run,benchmark,monitor,parse-hef,measure-power,tutorial,analyze-noise,compiler,params-csv,parser,profiler,optimize,tb,visualizer,har,join,har-onnx-rt,runtime-profiler,dfc-studio,help}
             ...
hailo: error: argument {fw-update,ssb-update,fw-config,udp-rate-limiter,fw-control,fw-logger,scan,sensor-config,run,benchmark,monitor,parse-hef,measure-power,tutorial,analyze-noise,compiler,params-csv,parser,profiler,optimize,tb,visualizer,har,join,har-onnx-rt,runtime-profiler,dfc-studio,help}: invalid choice: 'compile' (choose from 'fw-update', 'ssb-update', 'fw-config', 'udp-rate-limiter', 'fw-control', 'fw-logger', 'scan', 'sensor-config', 'run', 'benchmark', 'monitor', 'parse-hef', 'measure-power', 'tutorial', 'analyze-noise', 'compiler', 'params-csv', 'parser', 'profiler', 'optimize', 'tb', 'visualizer', 'har', 'join', 'har-onnx-rt', 'runtime-profiler', 'dfc-studio', 'help')
(hailo_virtualenv) hailo@ross1:/local/workspace$ hailo compiler yolov8s_pose_optimized.har 
[info] Current Time: 13:16:33, 01/21/25
[info] CPU: Architecture: x86_64, Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz, Number Of Cores: 16, Utilization: 0.1%
[info] Memory: Total: 125GB, Available: 123GB
[info] System info: OS: Linux, Kernel: 5.15.0-1073-azure
[info] Hailo DFC Version: 3.30.0
[info] HailoRT Version: 4.20.0
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler yolov8s_pose_optimized.har`
[info] Compiling network
[info] To achieve optimal performance, set the compiler_optimization_level to "max" by adding performance_param(compiler_optimization_level=max) to the model script. Note that this may increase compilation time.
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[error] Mapping Failed (allocation time: 12s)
No successful assignments: concat18 errors:
	Agent infeasible
concat17 errors:
	Agent infeasible
feature_splitter9 errors:
	Agent infeasible


[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed: No successful assignments: concat18 errors:
	Agent infeasible
concat17 errors:
	Agent infeasible
feature_splitter9 errors:
	Agent infeasible

Even the standard yolo gives me issues. If you could please provide a working example for the yolov8s_pose.onnx model the would go a long way to showing me where I’m going wrong.

I assume as there is a compiled version available in the zoo then it should be known.

I think it would be a good idea for you guys to document precisely how to compile all of the standard models in the zoo as a starting point for people. You could put these examples in the zoo linked from the model pages: hailo_model_zoo/docs/public_models/HAILO8/HAILO8_pose_estimation.rst at master · hailo-ai/hailo_model_zoo · GitHub

In fact, maybe even put the instructions here: Hailo Dataflow Compiler - Profiler Report

My colleague is also struggling to get any of this to work and he has a slightly different setup to me: Convert .onxx to .hef using the CLI