Support for GridSample operation

General Discussion
1

Is there planned support for GridSample Operation? Either in version 16 or 20.

If not, is there a way to bypass the lack of support? Can it be substituted using other operations?

Best regards,
Michał

2

Hey @quczer,

Right now, GridSample isn’t supported (and there’s no plan to add general support) because the operation doesn’t really fit with how our SDK works. That said, there are some workarounds depending on your case:

If the grid changes every inference (dynamic/learned):

  • You can’t directly convert GridSample to ONNX ops.
  • The best way is to handle the GridSample step on the host side—in Python, using PyTorch, OpenCV, or SciPy—before you send data to the Hailo-compiled model. Just remove GridSample from ONNX, do the warping in your app, and feed the model the already-warped input.

If the warp is always the same (static/simple):

  • You might be able to replace GridSample with standard ONNX ops:
    • Use Resize for simple upscaling/downscaling.
    • Use Pad, Slice, or Crop for padding/cropping.
    • For fixed affine transforms, you could combine Reshape, Transpose, etc.

So, bottom line:

  • For dynamic cases, move GridSample out of the graph and handle it in your host code.
  • For static warps, replace GridSample with equivalent ONNX operations.

Let me know your use case if you need more specific advice!

3

Hey @omria

As far as I understand it is an ONNX operation (see the link I provided).

I am trying to quantize a tracking neural network. It uses grid sample several times inside through deformable attention mechanism. TLDR; it predicts on which part of the image it should look and extracts corresponding feature maps to do next operations.

Because of that I am unable to extract this bit out of the compiled net, since I’d have to do up to 8 back-and-forth passes (many uses of deformable attention).

Do you see a way of tackling this problem other than substituting deformable attention w/ something else?

Thanks for your help!

4

Hello, sorry to bother you. How did you resolve the issue afterward? I’m also encountering this problem.

5

Hi! The “fix” was to just offload the grid sampling (and whole deformable attention too) to the CPU. It is a pain as I need to move weights between the CPU and Hailo, which is slow, but I didn’t find any better solution to this problem - grid sampling seems to be just not supported by the compiler.

6

Okay, thank you for your reply.