I am working on a real-time and accurate image matching pipeline using a Raspberry Pi 5 and Hailo-8. SuperPoint has been successfully converted and runs at around 300 FPS, but I have not been able to convert SuperGlue because the keypoint encoder produces shape/layout errors and the GNN attention layers use unsupported einsum operations. I am using fixed inputs with up to 500 keypoints and have also considered replacing einsum with MatMul and rewriting Conv1d layers, but shape errors and dimension mismatch still remain. Has anyone successfully converted the full SuperGlue model, or an accelerator-friendly ONNX version of it, for Hailo-8/Hailo-8L? Any export script, model, fork, or relevant experience would be greatly appreciated.
I was pushing SuperGlue through the DFC (3.33, hailo8 target) and got further than I expected, so let me share what works. Both problems you hit have a fix, and neither of them changes the maths of the model.
The keypoint encoder. The DFC does not read 1D tensors the way you would expect. The Conv1d layers get parsed on the wrong axis, so the compiler believes your 3 input channels are 500:
Invalid kernel shape for base conv layer base_conv1 (translated from /enc/enc.0/Conv).
Kernel features: 3 Input features: 500 Groups: 166
Rewrite the MLP with Conv2d 1x1 and feed (1, 3, 1, N) instead of (1, 3, N). The weights only need an unsqueeze(-1) to move across. That parses clean.
The attention. Swapping einsum for MatMul is the right idea, but the DFC only takes the matmul in the canonical MHA shape. Two things matter here. First, the transpose has to happen inside the matmul, not as separate permute calls before it — with explicit permutes I get Unexpected input shapes at matmul layer. Second, and this is the one that cost me most of the day, the N keypoints cannot sit on a flat axis. With (1, D, 1, N) the compiler fails with
Unexpected zero dimension in shape [-1, 0, 500, 256]
Give the keypoints a real spatial grid instead — 500 becomes 25x20 — and everything parses:
def forward(self, x, source): # both (1, 256, 25, 20)
q = self.proj[0](x).view(1, 4, 64, 500)
k = self.proj[1](source).view(1, 4, 64, 500)
v = self.proj[2](source).view(1, 4, 64, 500)
scores = (q.transpose(-2, -1) @ k) * 64 ** -0.5
prob = F.softmax(scores, dim=-1)
out = v @ prob.transpose(-2, -1)
return self.merge(out.reshape(1, 256, 25, 20))
There is one detail that is easy to miss because it fails silently. The original splits the 256 channels as c = d * num_heads + h, and this layout needs c = h * dim + d. So the q/k/v weights need their output rows permuted, and merge needs its input columns permuted the same way:
idx = torch.arange(256).view(64, 4).t().reshape(-1)
proj_w = original_proj_w.squeeze(-1)[idx].unsqueeze(-1).unsqueeze(-1)
proj_b = original_proj_b[idx]
merge_w = original_merge_w.squeeze(-1)[:, idx].unsqueeze(-1).unsqueeze(-1)
If you skip that step the model still compiles and still runs, it just returns rubbish.
With those changes I loaded the original superglue_indoor.pth weights into the rewritten model and compared it against the official one on the same inputs: max absolute difference on the score matrix is 5.3e-05 over a range of about ±32. So it is the same model, not an approximation. The full thing (18 attention layers, 73 MatMul) translates to 599 Hailo layers, and a single self+cross block with your dimensions compiles to 3.69 MB in one context.
Sinkhorn stays on the host. ReduceLogSumExp crashes the parser, so the optimal transport does not go on the chip. Not a big loss, it is cheap next to the GNN, but it means the HEF output is the raw NxN score matrix and you run the 100 iterations on the Pi.
Calibration matters more than usual here. With synthetic data the softmax over 500 tokens stays flat around 1/N and the optimiser aborts:
layer matmul3 does not support shift delta ... current range of input 0 is [0.002, 0.003]
With real SuperPoint descriptors that same layer gets [0.001, 0.334], which is a different world. So calibrate with real descriptor pairs from your own scenes, and expect to need quantization_param(force_range_in=...) on the matmuls that consume the softmax — the DFC prints the exact values it wants in the error.
I am still grinding through the quantisation of the full 18-layer stack, the shapes are not the problem at this point, it is the range forcing across 36 attention blocks. I will post the size and the accuracy against PyTorch once it finishes ..
Hope this helps!
Hi @Jesus_Royeth, Thanks for the tips! Using a very similar approach (switching to 4D layouts, keeping Sinkhorn off-chip), I finally managed to convert the model to HAR and successfully ran quantization using real scene data. The SNR is great, and the difference compared to the PT/ONNX model is very minimal.
The main problem now is not quantization or accuracy anymore. The real blocker is that Hailo is stuck for more than 48 hours at Finding the best partition to contexts while compiling the HEF, which means the graph is still too complex for the compiler to split efficiently, especially because of the deep attention/GNN stack and the large number of MatMul operations. Do you have the same problem or not?
Good news and bad news, and I think the bad one saves you time.
I hit exactly the same wall. My allocation also sat on “Finding the best partition to contexts” and it never came out — after 7h 34m it gave up with:
BackendAllocatorException: Compilation failed: Value doesn't fit in field (119)
Mapping Failed (Timeout, allocation time: 7h 34m 9s)
So it is not slow, it is not converging. I would kill that 48 hour run if you have not done it already ..
I ran a sweep to find where the limit actually is. Same model, same 500 keypoints, hailo8 with DFC 3.33:
| GNN layers | contexts | result |
|---|---|---|
| 1 (self+cross) | 1 | 3.69 MB  |
| 3 | 6 | 16.09 MB |
| 4 | 6 | 19.63 MB |
| 5 | 7 |  29 L4 memory cuts, max is 28 |
| 6 | — | Value doesn’t fit in field (119) |
| 9 (full model) | — | same, with the timeout |
The maximum is 4 GNN layers per HEF. At 5 it misses by a single memory cut: going from 6 to 7 contexts it asks for 29 and the cap is 28. From 6 layers up the error changes and it is no longer about resources, it is a field overflow in the compiler — the same one that kills the full model.
So the 9 layer model splits into three HEFs (3+3+3 is balanced), chained by passing desc0/desc1 from one into the next. The GNN is a homogeneous stack, so the cut needs no graph surgery.
The part that unblocks you today: you do not need the HEF to know whether the quantisation is good enough. The DFC runs the quantised model from the HAR you already have, and it is bit-faithful to what the silicon would do:
from hailo_sdk_client import ClientRunner, InferenceContext
runner = ClientRunner(har="your_model.har")
with runner.infer_context(InferenceContext.SDK_QUANTIZED) as ctx:
out = runner.infer(ctx, feed) # feed is NHWC
I ran that on the full model against PyTorch, 16 real pairs from the freiburg sequence, same host-side Sinkhorn on both sides:
matches PyTorch : 4666
matches quantised : 4210
recall : 70.8%
precision : 78.5%
Pearson r (matrix) : 0.759
You keep about 90% of the match volume and roughly 70% land on the same keypoint. Good enough for RANSAC on pose or homography, but it is a real drop. Two caveats: the measurement is on the unsplit model, so treat it as a ceiling — the three HEF version adds one conversion per boundary and I have not measured that yet. And it comes from the quantised emulator, not from hardware, I do not have the board here.
One thing I got wrong, so you do not spend time on it: I assumed the force_range was too tight and clipping the value tensor. I measured the real ranges with hooks and that was not it — the softmax reaches 1.0 (trained attention is nearly one-hot) and the value tensor sits in [-15.8, 13.3]. Widening the clip from ±8 to ±16 moved recall by 0.2 points, which is noise. The range is not the bottleneck.
Underneath it is the 8-bit matmul stacked over 18 attention layers. On this generation the matmul only takes a8_w8, a8_w8_a8 and a8_w8_a16, so there is no 16-bit knob for attention, and the error compounds layer after layer. I tried QAT to see how much it recovers and it died on memory (26.8 GB of RSS on a 31 GB machine), so that one is still open.
So, answering your original question: yes, SuperGlue can be deployed on Hailo-8, just not as a single HEF. The recipe that works:
- Rewrite the keypoint encoder with
Conv2d1x1 and the attention in the canonical MHA shape, with the keypoints on a spatial grid — details in my previous message, including the channel reordering that fails silently. - Quantise with real descriptors from your own scenes and
force_range_inon the matmuls that consume the softmax. This gets you to the HAR with no trouble. - Cut it into three HEFs of 3 layers and chain them. Compiling it whole does not converge, no matter how long you wait.
- Sinkhorn and the final matching on the Pi CPU.
Expect around 70% of the PyTorch matches. RANSAC will estimate pose fine with that, but if your application needs the exact correspondences then the 8 will fall short and it is worth looking at the 15/10, where the compiler handles attention much better.