Parse Transformer Decoder

General
1

Hello, I am trying to convert a pytorch Transformer Decoder from onnx to Hailo8l.

However when I try to parse it in the dfc, I get the following errors:
Parsing failed. The errors found in the graph are:
UnsupportedOperationError in op /decoder/GatherElements_1: GatherElements operation is unsupported
UnsupportedReduceMaxLayerError in op /decoder/ReduceMax: Failed to create reduce max layer at vertex /decoder/ReduceMax. Reduce max is only supported on the features axis, and with keepdim=True
UnsupportedOperationError in op /decoder/TopK: TopK operation is unsupported
UnsupportedOperationError in op /decoder/GatherElements: GatherElements operation is unsupported
Please try to parse the model again, using these end node names: /decoder/Add, /decoder/enc_output/norm/LayerNormalization, /decoder/enc_score_head/Add.

Exporting the Encoder worked fine, so I do not understand why converting the Decoder does not work.
I am new to Hailo, so maybe I am not seeing something obvious.

Thanks for any help :slight_smile:

2

Hey @Byte ,

Your ONNX model’s decoder conversion is failing due to several unsupported operations on Hailo-8. Here’s how to resolve these issues:

Unsupported Operations and Their Replacements:

  1. GatherElements → Index Select
    • Why: GatherElements isn’t supported on Hailo hardware
    • Replacement maintains similar functionality while using supported operations
# Before: Using GatherElements
gathered = torch.gather(tensor, dim=-1, index=indices)
# After: Using Index Select
indexed = torch.index_select(tensor, dim=-1, index=indices)
  1. TopK → ArgMax with Masking
    • Why: TopK operation isn’t supported on Hailo
    • ArgMax provides similar functionality when you only need the highest value
# Before: Using TopK
top_values, top_indices = torch.topk(tensor, k=1, dim=-1)
# After: Using ArgMax
max_indices = torch.argmax(tensor, dim=-1, keepdim=True)
  1. ReduceMax → ReduceMax with keepdims
    • Why: ReduceMax is only supported on features axis with keepdims=True
    • Simple modification to existing operation
# Before: Standard ReduceMax
max_values, _ = torch.max(tensor, dim=1)
# After: ReduceMax with keepdims
max_values, _ = torch.max(tensor, dim=1, keepdim=True)

Alternatively, if modifying the model is too complex, you can truncate the unsupported layers:

hailo parser onnx decoder.onnx --end-node-names /decoder/Add /decoder/enc_output/norm/LayerNormalization /decoder/enc_score_head/Add

Let me know if you need help implementing any of these changes or have questions about specific operations!