Hailo Compiler Crash when Compiling Upscaling Network

General
, ,
1

So this is really more of a bug report than anything else, but I’m not really sure where else to share it.

I’ve been trying to compile the model in the attached HAR file for the Hailo 8, and I can get parsing and optimization to work, but the final compilation to HEF always fails. The compiler consistently crashes with a weird error:

[info] No GPU chosen and no suitable GPU found, falling back to CPU.
[info] Current Time: 13:01:14, 10/15/25
[info] CPU: Architecture: x86_64, Model: 13th Gen Intel(R) Core(TM) i7-13700K, Number Of Cores: 24, Utilization: 5.3%
[info] Memory: Total: 31GB, Available: 11GB
[info] System info: OS: Linux, Kernel: 6.9.3-76060903-generic
[info] Hailo DFC Version: 3.33.0
[info] HailoRT Version: Not Installed
[info] PCIe: No Hailo PCIe device was found
[info] Running `hailo compiler --hw-arch hailo8 ../git/foundation_model/plantsam_bc2_multi_modal_full_renamed_quantized_model.har`
[info] Compiling network
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
/usr/include/c++/9/bits/stl_vector.h:1120: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::front() [with _Tp = long unsigned int; _Alloc = std::allocator<long unsigned int>; std::vector<_Tp, _Alloc>::reference = long unsigned int&]: Assertion '__builtin_expect(!this->empty(), true)' failed.

[error] Failed to produce compiled graph
[error] BackendAllocatorException: Compilation failed with unexpected crash

I’ve been flummoxed by this for awhile now, but by using the remove_nodescript feature I think I’ve finally narrowed it down to a single upscaling head at the end of the model. For reference, the attached script, which disables all the nodes in this head, fixes the issue I’m having. I just wanted to report this as a likely bug in the compiler. Thanks!

2

Hey @Daniel_Petti ,

Welcome to the Hailo Community!

Thanks for sharing this!
I have reported this bug , hopefully will be handled soon!

1 Like
3

Thanks, I just wanted to follow up that I have since tried compiling a version of the model where I completely removed the problematic head. I tried removing it both from the original ONNX and by setting the end nodes during the parsing stage, but oddly, neither way fixes the original problem. Both of these new models crash the compiler with the exact same error. Therefore, the cause might not be as simple as I assumed it was.