DFC 3.34.0 / Hailo-8L — Compiler SIGABRT on fixed selection-matrix matmuls

General Discussion
, , ,
1

Environment:

  • Hailo DFC 3.34.0
  • Target: hailo8l
  • Ubuntu 24.04 / WSL2, Python 3.10

Model: A 6-block, 768-dim, 12-head RoPE-based Vision Transformer encoder (attention-only, standard pre-norm blocks, ~7M params/block). ONNX export verified numerically correct against the original PyTorch model (max relative error ~1e-6 across the pipeline). Translation and INT8 quantization succeed reliably on every graph variant described below — only allocation/compilation fails.

Failure 1 — Compiler SIGABRT (fatal signal 6) on fixed selection-matrix matmuls

Our graph uses fixed 0/1 selection-matrix matmuls (shapes like [1,512,256], [1,516,512]) to scatter/assemble tokens (query + visual + register tokens into one sequence) — this replaced Concat ops that were separately failing with Agent infeasible at the same boundary. With this matmul-scatter approach, the native compiler process aborts with SIGABRT during allocation, with no structured error message returned to the Python API (BackendAllocatorException: Compilation failed with unexpected crash). Kernel trace confirms fatal signal 6 in the compiler process. Reproduces even on a minimal isolated graph containing only the token-assembly step (no transformer blocks at all) — so it’s specific to this op pattern, not model size. Quantization of this same graph completes successfully; only runner.compile() crashes.

Failure 2 — auto_spatial_reshape / Agent infeasible on bare [seq, channels] sequence boundaries

Independently, when a [1, 516, 768] sequence activation (register+query+visual tokens, pre-head) sits at a graph input or output edge — e.g., when compiling a subset of transformer blocks as their own graph — allocation fails with Agent infeasible on auto_spatial_reshape_from_input_layer1_to_conv2... (and the mirrored error at the output boundary). We attempted the documented format_conversion(layer, hailo_rgb_to_tf_rgb / tf_rgb_to_hailo_rgb) workaround (as suggested by the compiler’s own hint text) on both input and output edges; this did not resolve it — the format_conversion layers themselves come back with Agent infeasible. Reproduces on a single isolated attention block and on a 6-block-only subgraph (no stem).

What does work: the full single-graph encoder (stem + 6 blocks + head as one ONNX graph, so all [seq,768] tensors are internal, never at a graph edge) translates and quantizes successfully, but allocation for the full graph separately times out (native 1-hour resolver watchdog, ~1484/1486 iterations reverting on cluster mapping in context_0) rather than crashing — suggesting the graph is right at or past the edge of what a single-context/limited-context mapping can place.

Questions:

  1. Is there a known limitation or workaround for large (>~100K element) fixed selection-matrix matmuls used for token scatter/gather — is there a supported alternative op pattern for this kind of static index-based token assembly?
  2. What is the correct way to expose a [seq, channels] transformer sequence activation (not an image-shaped tensor) at a Hailo graph edge, if format_conversion isn’t sufficient? Is there a required intermediate reshape/layout op we’re missing?
  3. Any guidance on why the SIGABRT occurs without a diagnostic message — is there a way to get more verbose native-compiler error output from hailo_sdk_client?

Happy to share the minimal repro ONNX files / HARs if useful.

2

Hi @Vishal_Ganpisetti :

I’m on DFC 3.33.0, not 3.34, but I isolated this with a small synthetic model and it looks like both of your symptoms come from the same root cause: the row count of the selection matmul output.

I swept the row count of a [1,N,256] selection matmul, everything else fixed:

N = 500, 508, 513, 516, 524  -> fails (silent crash or "Agent infeasible")
N = 512, 520, 528            -> compiles clean

The pattern is N % 8 == 0. Your first matmul ([1,512,256]) already satisfies that, which is why it compiled fine on its own. Your second one ([1,516,512]) doesn’t, 516 isn’t a multiple of 8, and that’s what triggers the crash — not the chaining, not the total size.

With your exact shapes chained directly I get your crash character for character:

[error] Failed to produce compiled graph
hailo_sdk_client.sdk_backend.sdk_backend_exceptions.BackendAllocatorException: Compilation failed with unexpected crash

allocator.log stays empty and nothing in the logs mentions a signal, so the native compiler process seems to die before it can write its structured error report. No coredump on my side either (ulimit -c was 0), so I can’t confirm the exact signal number.

What did compile: pad the selection matrix rows to the next multiple of 8 (516 → 520), compile that matmul as its own standalone graph, and crop the extra rows back to 516 in the host after inference, with plain numpy slicing. A few things I tried that don’t work, so you don’t have to burn time on them:

  • Putting the crop back inside the graph as a Slice node fails too (auto_spatial_reshape_from_matmul1_to_slice1: Agent infeasible).
  • Chaining the padded matmul directly after the first one, in the same graph, still fails — with a third, different error this time (Node matmul1 is not supported as weights node for matmul2).
  • allocator_param(enable_partial_row_buffers=disabled) and performance_param(compiler_optimization_level=0) don’t move it either.

So the working shape is: two separate HEFs, not one graph, with the pad/crop happening in host code around them.

I think this also explains your second symptom. Exposing [1,516,768] at a graph boundary and hitting auto_spatial_reshape is probably the same 516-isn’t-a-multiple-of-8 issue, not a separate problem with exposing sequence activations at edges.

I haven’t tried this at the scale of your full 6-block encoder with RoPE, so I can’t tell you it holds end to end .. but padding every selection matmul to a multiple of 8 and keeping them as separate HEFs seems like the right place to start.