Has anyone successfully deployed a neural single-object tracker (.hef) on Hailo-8L?

[Abhay_Payadi]

Hi everyone,

I’m working on a single-object tracking application on Hailo-8L and I’m looking for a neural tracker that can run almost entirely on the Hailo accelerator.

My setup

• Hailo-8L
• Hailo DFC 3.34.0
• HailoRT
• Single object (car tracking)
• Linux / C++
• Detector: YOLO (already running successfully on Hailo)

What I’ve already tried

I successfully exported and parsed the NanoTrack backbone.

However, the NanoTrack head (DepthwiseBAN) fails during parsing because of the dynamic depthwise cross-correlation layer.

I’ve also looked into SiamRPN/PySOT and understand they rely on similar Siamese correlation operations that are difficult for the Hailo compiler.

What I’m looking for

I’m not looking for ByteTrack, DeepSORT, or CPU-based tracking.

I’m specifically looking for a neural single-object tracker whose entire network (or nearly the entire network) can be compiled into a .hef and run on Hailo.

Examples I’m curious about:

• LightTrack
• SiamFC++
• MixFormer-lite
• OSTrack-lite
• HiFT
• ROMTrack
• Any custom CNN-based tracker

Has anyone successfully deployed a neural tracker like this on Hailo?

If yes, could you share:

• Tracker name
• GitHub/project
• Whether it compiles to .hef
• Any modifications required

Thanks!

[Andrew_Jewell]

Yes, I got a full neural single-object tracker compiling to a single-context .hef on Hailo-8L. The trick was replacing the dynamic correlation, not dropping the Siamese idea.

Your diagnosis is right. DepthwiseBAN, SiamRPN, and SiamFC++ all do F.conv2d(search, template_features_as_weight), so the template becomes a conv kernel generated at runtime. A static-graph compiler can’t lower that because the weights depend on the input.

The fix is to express template-to-search matching with static ops. I built a small custom CNN plus a cross-attention tracker, where search locations are queries and template locations are keys/values (Q·Kᵀ → Softmax → ·V). The correlation comes out as Conv 1×1 + Reshape + Transpose + MatMul + Softmax, all standard and all DFC-supported, so the whole network parses and compiles. It’s also a stronger matcher than xcorr.

DFC-verified results (Hailo DFC 3.33.1, --hw-arch hailo8l):

• Parser translates the entire graph to end nodes: cls, box, nothing left on CPU.

• Single-context .hef, about 1.9 MB, 4 clusters, roughly 50% compute utilization.

• hailo profiler estimate is about 2,964 FPS at 0.75 ms latency.

• About 540K params. Template 1×3×128×128, search 1×3×256×256, out to targetness 1×1×32×32 plus ltrb box 1×4×32×32 (anchor-free). Profiler report attached.

Three things that actually mattered to get it through the DFC:

1. Use single-head attention. Multi-head reshapes the head dim into the batch axis and does a batched MatMul, and the DFC silently cut the graph at the correlation. Single-head keeps everything at batch = 1 and the parser goes end to end. If your parse “succeeds” but the recommended end-nodes land mid-net, this is why.

2. Export SiLU as Sigmoid plus Mul, not a single Silu/Swish node. The onnx.checker rejects the non-standard op (No Op registered for Silu).

3. Export Reshape with the shape as an input tensor, not an attribute (opset 13+). Otherwise the checker throws input size 1 not in range [2,2].

Also: fixed input shapes, fold BN, anchor-free head (a couple of 1×1 convs for cls and box), quantize to int8.

On your candidate list: OSTrack-lite, MixFormer-lite, and HiFT are attention-based, so correlation is already static MatMul/Softmax. Best starting points, just watch the multi-head batch-reshape. SiamFC++ and SiamRPN still use xcorr, same wall. LightTrack is mostly CNN, but check the head and rewrite any pointwise correlation as a MatMul.

Happy to share more on the attention-correlation block if it’s useful, so reach out if you want it.

[abhijeet_kale]

could you share the .hef for this?

[Abhay_Payadi]

Thanks for the detailed explanation—this is extremely helpful. It confirms what I was running into with dynamic correlation in Siamese trackers.

I had a few questions if you don’t mind sharing:

1. Is your backbone purely CNN (e.g. MobileNet/ShuffleNet), or does it include lightweight Transformer blocks before the attention-based matching?
2. Could you share the structure of the single-head attention correlation block (or a simplified PyTorch/ONNX implementation)? I’m particularly interested in how you reshape the template/search features before the MatMul.
3. Is the attention computed over the full spatial feature map, or do you reduce the feature dimension first with a 1×1 convolution?
4. Approximately what feature map resolution are you using before attention (e.g. 16×16, 32×32)?
5. Were there any other ONNX operators that required modification besides SiLU, dynamic Reshape, and multi-head attention?
6. If it’s possible, could you share either a network diagram, an ONNX graph, or even just the attention block? That would be incredibly useful for understanding how you kept the graph fully static for the DFC.

Did you train this architecture from scratch, or was it adapted from an existing tracker such as NanoTracker,OSTrack, MixFormer, or HiFT?

If possible can you share the .onnx or .hef file , it would be helpful.

Thank You

[Abhay_Payadi]

Hi,

I followed your suggestion to replace NanoTrack’s dynamic depthwise cross-correlation with a static attention-based module that is Hailo-friendly.

Changes made

• Replaced DepthwiseXCorr with a cross-attention module:

  • 1×1 Conv projections (Q, K, V)
  • MatMul
  • Softmax
  • MatMul
  • 1×1 output projection

• Removed the dynamic grouped convolution used for cross-correlation.

• Kept the rest of the NanoTrack V3 architecture unchanged.

Current status

• PyTorch inference works correctly.

• Model outputs:

  • cls: [1, 2, 15, 15]
  • loc: [1, 4, 15, 15]

• ONNX export succeeds.

• onnx.checker.check_model() passes.

• hailo parser succeeds without any unsupported operator errors.

• HAR file is generated successfully.

However, the optimization step fails.

DFC Version

• DFC 3.34.0
• Target: Hailo-8L

Optimization command

hailo optimize \
    nanotrack_hailo.har \
    --use-random-calib-set \
    --hw-arch hailo8l

Error

The optimizer crashes inside:

input_features_defuse.py

initial_split_size = pred_output_shape[-1] // num_splits

ZeroDivisionError: integer division or modulo by zero

This happens even when using --full-precision-only, so it appears before quantization.

Additional observations

The exported ONNX outputs are:

cls : [1, 2, 15, 15]
loc : [1, 4, 15, 15]

After parsing, the extracted HN reports:

conv69 (cls_logits):
shape = [-1, 13, 78, -10]

conv84 (bbox_pred):
shape = [-1, 13, 78, -20]

These correspond to the final ONNX output nodes:

/ban_head/cls_logits/cls_logits.0/Conv
/ban_head/bbox_pred/bbox_pred.0/Conv

The parser completes successfully, but the output tensor metadata in the generated HN appears inconsistent with the ONNX model. The optimizer then crashes during input_features_defuse.

Could you please let me know if this is a known limitation or bug in DFC 3.34, particularly for transformer-style or attention-based multi-input networks? I’d also appreciate it if you could suggest any workaround or recommend a newer DFC version if this has already been fixed.

I can share the ONNX model, HAR file, and full optimization logs if needed.

[Andrew_Jewell]

Nice progress, and thanks for the detailed writeup. The good news is this almost certainly isn’t attention being unsupported in DFC 3.34. The real problem is upstream of the optimizer, and the smoking gun is right there in your HN dump.

Look at these:

conv69 (cls_logits): shape = [-1, 13, 78, -10]
conv84 (bbox_pred):  shape = [-1, 13, 78, -20]

Those should map to your ONNX outputs [1, 2, 15, 15] and [1, 4, 15, 15]. They don’t, not even close. The parser reported “success” but it mis-inferred the shapes somewhere in the attention block and produced garbage output metadata with negative dims. The optimizer crash is just the downstream symptom: input_features_defuse computes num_splits from that last dim, gets a bogus value, and you hit the divide by zero. Chasing the optimizer or --full-precision-only won’t help because the tensor is already wrong before optimization starts. That also matches your own observation that the HN metadata is inconsistent with the ONNX.

So the thing to fix is the export, so the parser infers clean shapes through the attention. The usual causes, in the order I’d check them:

1. A -1 in one of your reshapes. The [-10] and [-20] in the HN look exactly like an unresolved dynamic dim propagating through. Bake every reshape target to explicit integers. Don’t use .view(1, C, -1) or anything with a -1, compute the real numbers (H*W = 225 here) and hardcode them.

2. The flatten to 3D for the MatMul. When you reshape [1, C, H, W] down to a 3D tensor for Q·Kᵀ, the parser’s shape tracking in NHWC gets confused. Keep everything 4D and batch = 1 through the whole correlation, and only reshape with baked constant shapes. This is the same batch-axis issue that bit me, it’s just showing up as a shape corruption instead of a mid-net end-node cut.

3. Run the ONNX through onnx-simplifier before parsing. That folds the constant shape math and kills most of the phantom dynamic dims. In a lot of these cases that alone gets you a clean parse.

4. Keep opset at 13 or higher and export reshape shapes as input tensors, not attributes.

After you re-export, re-parse and actually inspect the HN shapes at the attention output, not just at the end nodes. The first layer where the shape stops matching your ONNX is your culprit op. Once cls and loc come out as the right 4D shapes in the HN, the defuse crash should go away on its own.

One more thing: I wouldn’t jump to a DFC upgrade yet. If the ONNX still exports ambiguous shapes, a newer version will trip on the same thing.

If you want, share the simplified ONNX and the parse log and I’ll take a look at where the shapes drift. Reach out if that would help.

[Abhay_Payadi]

I updated the model to remove all dynamic reshapes from the attention block. The exported ONNX no longer contains Shape nodes inside corr_dw_cls or corr_dw_reg; all reshape targets are constants. I also ran onnxsim, and onnx.shape_inference reports valid static shapes throughout the attention block (e.g. MatMul: [1,225,16], Softmax: [1,225,16], Reshape: [1,96,15,15], out_proj: [1,96,15,15]). Despite this, the Hailo parser still generates HN output shapes [-1,13,78,-10] and [-1,13,78,-20], and the optimizer crashes as before. This suggests the shape corruption occurs inside the parser rather than in the exported ONNX. Is there a known limitation with this attention pattern, or is there a parser debug option that can identify where the HN shape becomes invalid?

Can you also let me know about your model , which tracker did you work on and how did you debug the issue . Soo i can try with different tracker and get an idea (if its not nano tracker).

[Abhay_Payadi]

The ONNX graph uses Concat(axis=1) (channel concat in NCHW) between corr_pw_* and corr_dw_*. onnx.shape_inference reports both inputs as [1,192,15,15] and [1,96,15,15]. However, in the generated HN, the parser represents the inputs as [-1,1,63,192] and [-1,15,15,96] and rewrites the operation to concat_axis = spatial_w, producing [-1,1,78,192]. All subsequent tower convolutions inherit invalid shapes (-192, -384, etc.), and the final outputs become [-1,13,78,-10] / [-1,13,78,-20]. The attention branch itself parses correctly; the first incorrect metadata appears at the concat between the pointwise and attention branches.

[Andrew_Jewell]

That concat dump is the most useful thing posted in this thread — it pins the problem precisely. But I don’t think the concat is where it starts.

Read the two inputs as the parser reports them:

• [1,96,15,15] → [-1,15,15,96] — correct NCHW→NHWC.

• [1,192,15,15] → [-1,1,63,192] — not the NHWC of that tensor. It should be [-1,15,15,192].

So the corr_pw_* tensor is already corrupt when it arrives at the concat. The concat then has two tensors it cannot channel-concat (H,W of 1,63 vs 15,15), so it falls back to concatenating along W — and 63 + 15 = 78, which is exactly the 78 that appears in [-1,1,78,192] and then rides through every downstream shape into [-1,13,78,-10] and [-1,13,78,-20]. The negative channel dims are just the tower convs deriving output channels from an already-broken channel count.

That 78 being precisely 63+15 is what tells you this is a layout fallback rather than arithmetic noise. So the first bad metadata isn’t at the concat — it’s upstream in the pointwise branch. The concat is merely the first op that has to reconcile two layouts, which is why that’s where it becomes visible.

The question worth answering before anything else: how is the pointwise correlation implemented? In most Siamese trackers (NanoTrack included) the correlation is F.conv2d(search, kernel=template, groups=C) — a Conv whose weights are an activation, not an initializer. ONNX exports that as a Conv with a non-constant second input, and the parser expects Conv weights to be constants. Dynamic-kernel / dynamic-group conv is exactly where layout inference will hand you a plausible-looking but wrong [-1,1,63,192]. The fact that corr_dw_* parses cleanly while corr_pw_* doesn’t is consistent with the two being constructed differently — worth diffing how each is built.

To localise it exactly:

1. Bisect with the parser. translate_onnx_model(..., start_node_names=[...], end_node_names=[...]) — cut just before the concat, dump the HN, then walk the end node backward through corr_pw_* one op at a time. The first node whose HN shape stops matching onnx.shape_inference is your culprit. Based on the above, expect it well upstream of the concat.

2. Diff the HN per layer rather than reading the final outputs — runner.get_hn(), or save_har() and pull the .hn JSON out of the archive. Shape-by-shape against shape_inference finds the drift point in minutes.

3. Pin the input with net_input_shapes={...} so you aren’t also fighting a batch/layout ambiguity, and set the hailo_sdk_client logger to DEBUG.

If it is the dynamic-kernel correlation, the pragmatic move is to cut the graph there: compile the backbone/branches separately and do the correlation on the host. It’s a handful of cheap ops at 15×15×{96,192} and it sidesteps the parser entirely. If your template is fixed at build time, bake it in as a real initializer — the conv becomes an ordinary static conv and parses without complaint.

On your question: Siamese correlation tracker, same family. The method that matters is step 1/2 above — stop reading the end nodes and diff the HN layer by layer against shape_inference until you find the first op where they part company. Reasoning backward from the final output shape will send you chasing the optimizer every time; the drift point is always earlier than it looks.

Post the simplified ONNX and the parse log if you want another set of eyes on where the shapes go.

[Andrew_Jewell]

Quick follow-up — I think the clean shape_inference through the attention block is a red herring, and it’s worth seeing why before you file it as a parser bug.

Look at your own concat dump again: the two inputs are corr_pw_* → [-1,1,63,192] and corr_dw_* → [-1,15,15,96]. The corrupt one is the pointwise branch (192-ch, now 1×63). The attention branch (corr_dw_*, 96-ch) reads correctly — [-1,15,15,96] is the right NHWC. So the block you just made static (the attention) was never the one breaking; cleaning its reshapes and re-running shape_inference on it doesn’t touch the tensor that’s actually wrong, which is why the HN output didn’t move.

And you’ve got the parser part right: shape_inference passing end-to-end doesn’t rule out a parser problem, because the parser runs its own layout inference from scratch and doesn’t consume the ONNX shape results — a graph that’s perfectly valid to ONNX can still be re-inferred wrong here. It’s just re-inferring corr_pw_* wrong, not the attention.

So before “known parser bug,” pin it with the bisect: translate_onnx_model(..., end_node_names=[<last node of the corr_pw_ branch, before the concat>]), dump the HN, and walk that branch backward one op at a time against shape_inference. The first op where the HN shape stops matching is the drift point — and it’ll be in corr_pw_*. That node is almost always either (a) a Reshape/Transpose in the pointwise branch that permutes and the parser mis-tracks, or (b) corr_pw_ being constructed differently from corr_dw_. Diffing how the two branches are built in your export script usually makes it jump out.

Two things end it regardless of the exact op:

• Give the pointwise branch the same explicit [1,C,15,15] + Transpose treatment you gave the attention branch, so both concat inputs are unambiguous 4D NCHW before the merge; or

• if corr_pw_ is still a correlation (dynamic kernel), cut it out and do that one op on the host — it’s tiny at 15×15 and sidesteps the layout inference entirely.

On your other question — it’s not really a tracker-selection issue, so swapping NanoTrack for a different one won’t dodge it. Any Siamese with a two-branch template/search merge hits the same layout-reconciliation step; whether it parses comes down to how you build and merge those two branches, not which tracker they’re from. Mine was a same-family Siamese, and the only debugging that moved things was that layer-by-layer HN-vs-shape_inference diff — reasoning back from the final -10/-20 output points you at the optimizer every time, but the drift is always earlier, in one branch.

Post the corr_pw_ sub-graph (the branch output + how it’s built) and the parse log at DEBUG and I’ll take a look.

---

[Abhay_Payadi]

Hi,

I did some more debugging over the weekend and narrowed the issue down further.

• The attention (corr_dw ) branch parses correctly. Its output appears in the HN as [-1,15,15,96] , matching the expected layout.
• The problem starts in the corr_pw branch before the concat.
• ONNX shape inference reports:
  • CA_layer/Mul_output_0 → [1,64,16,16]
  • conv.0 (depthwise 2×2, groups=64) → [1,64,15,15]
  • conv.3 → [1,64,15,15]
• However, in the generated HN:
  • ew_mult15 (mapped to CA_layer/Mul ) has output [-1,1,64,256] , which seems consistent with Hailo’s flattened internal representation.
  • The very next layer (conv51 , corresponding to corr_pw_cls/conv/conv.0 ) changes to [-1,1,63,192] .
• At the concat, the other branch is still [-1,15,15,96] , so the parser concatenates along width, producing [-1,1,78,192] (63 + 15 = 78), and all downstream shapes become incorrect.

This seems to indicate that the first divergence happens at the depthwise Conv2d(kernel_size=2, groups=64) immediately after CA_layer/Mul , rather than at the concat itself.

Does this look like a known parser/layout inference issue? Also, is there a recommended way to bisect the parser (e.g., translating only a subgraph or specifying start/end nodes) so I can isolate the first operation where the HN shape diverges from the ONNX shape?

Thanks!

[Abhay_Payadi]

Hi,

Thank you for your guidance so far.

After further debugging, we confirmed that the grouped convolution was not the issue. We modified the model to use a standard convolution (groups=1), regenerated the ONNX and simplified ONNX, and verified that the Hailo parser also preserves groups=1.

However, we observed that the parser fuses the MatMul + Reshape sequence into a single matmul operation. As a result, the expected spatial tensor (64×16×16) is kept as a flattened representation (64×256), and the subsequent spatial convolution no longer behaves as expected. At this point, I think this is an architectural incompatibility rather than a simple parser configuration issue.

Rather than spending more time on NanoTrack, I’m planning to evaluate another lightweight Siamese correlation tracker, such as LightTrack.

Before I move on, I had one last question:

• Have you worked with or deployed any other Siamese correlation-based trackers on Hailo (apart from NanoTrack)?
• If yes, which tracker did you use (e.g., LightTrack, SiamFC, SiamRPN, or a custom tracker)?
• How did you approach debugging parser/compiler issues when porting the model? Were there any architectural changes you found necessary to make the tracker Hailo-friendly?

Any recommendations or lessons learned would be greatly appreciated before I start integrating another tracker.

Thank you again for your support and guidance.

[Andrew_Jewell]

Hi Abhay, your bisection is solid and you’ve landed on the real issue. Let me reframe it because it generalizes to every Siamese tracker, which answers your last questions too.

The root cause isn’t the groups=64 conv or the concat. It’s the MatMul → Reshape → spatial-Conv pattern. Hailo’s compiler runs a fixed spatial (NHWC) dataflow. When a MatMul produces a matrix-shaped tensor and a Reshape turns it back into a spatial map that a downstream Conv consumes, the compiler keeps the tensor in its flattened matmul form (your 64x256) because it can’t reconstruct the spatial layout the conv expects. That’s why the shape diverges at the conv after the matmul, not at the concat. Fusing MatMul+Reshape is intentional and not something you can config away, since the fused op simply has no spatial-conv-compatible mapping.

There are two sub-cases, and the fix differs:

If that MatMul is a static projection (learned weight matrix, e.g. a channel mixing / 1x1-equivalent), rewrite it as an actual 1x1 Conv2d in the model. A 1x1 conv is a per-pixel channel matmul but stays in spatial form, so it composes cleanly with the following conv and the flatten disappears. This is the single most useful “make it Hailo-friendly” refactor for this pattern, worth trying before you abandon the model.
If that MatMul is the cross-correlation itself (template features correlated against search features), you’ve hit the hard wall. The correlation is a dynamic-weight operation: one operand (the template) acts as the kernel. Hailo hardware only supports static, compile-time weights, so a dynamic correlation cannot run on the NPU at all. NanoTrack, SiamFC, SiamRPN(++), LightTrack all share this exact bottleneck (DW-XCorr / pointwise-XCorr). Switching trackers won’t get you past it.
So the robust, tracker-agnostic pattern is: put the two backbone feature extractors on the NPU (static weights, that’s the expensive part), and do the correlation plus the small head on the host CPU. Cross-correlating two ~15x15x64 feature maps is trivially cheap on CPU. Cut the ONNX right before the correlation, compile that as your HEF, and stitch the rest in your app code. This works identically for LightTrack, SiamFC, SiamRPN. The tracker choice then comes down to backbone accuracy/latency, not Hailo compatibility.

On bisecting the parser (this is the useful bit for isolating the divergence):

translate_onnx_model(…, start_node_names=[…], end_node_names=[…], net_input_shapes={…}) lets you translate an arbitrary subgraph. Walk end_node_names forward node by node until the HN shape first diverges from ONNX, which pinpoints the offending op precisely.
After each translate, runner.save_har() and inspect the parsed graph / layer shapes to compare against ONNX shape inference.
Run the model through onnx-simplifier first. It sometimes collapses reshape/transpose chatter into a Hailo-friendly form. It won’t remove a true dynamic correlation, but it clears up false positives.
Cutting at end_node_names right before the correlation is also exactly how you’d generate the “backbone-only” HEF for the split above.
Architectural lessons that transfer: avoid Reshape between matrix and spatial forms around convs, keep spatial ops purely convolutional (prefer 1x1 conv over MatMul+Reshape for channel projections), and treat any dynamic-weight op (cross-correlation, or attention where both operands are activations feeding a conv) as a host-side stage from the start. Design the graph so the NPU boundary lands on the static-weight backbone and everything dynamic lives on the CPU.

Hope that saves you the LightTrack detour. Happy to look at the specific cut point if you share where in the graph your correlation sits.