Custom 3D NMS Layer

General
,
1

Hi!

I am currently working on deploying a 3D object detection model (PointPillars) using the Hailo-8 accelerator in conjunction with a Raspberry Pi, and I am striving for a fully self-contained solution that does not rely on GPU (i.e., no CUDA). The ultimate goal is to perform real-time inference entirely on the Hailo device and the accompanying CPU.

So far, I have successfully converted and deployed almost the entire PointPillars model to run on Hailo. The preprocessing stage runs efficiently on the CPU, and the main backbone and detection head have been compiled and executed on Hailo using the standard flow. The only remaining component that currently blocks full deployment is the post-processing stage, specifically the 3D Non-Maximum Suppression (NMS) operation.

At present, I am executing the 3D NMS on the CPU, but it represents a significant bottleneck in terms of latency. I have been exploring the possibility of including this step as a layer at the end of the model graph. I noticed that Hailo supports an optimized NMS implementation for specific models like SSD and YOLO, and based on this, I was able to successfully integrate the SSD-style NMS layer into my graph.

However, these built-in NMS layers are designed for 2D bounding boxes, and they do not account for properties critical in 3D detection, such as bounding box rotation (i.e., yaw angle).

Given this, I would like to ask:

  1. Is it possible to design a custom 3D NMS layer (or modify the existing NMS operator) and have it executed on the Hailo device?
  2. Does Hailo offer any flexibility to extend or adapt the internal NMS implementation to support rotated boxes or non-axis-aligned suppression?
  3. If so, could you provide guidance or documentation on how to implement and integrate such a custom layer?

I am open to simplifying the suppression logic if needed—for instance, projecting 3D boxes into BEV and suppressing based on 2D overlap, but I do need to take rotation into account.

Any information you can provide on how to proceed with integrating a custom or modified NMS solution that executes on Hailo would be highly appreciated.