YOLOv5s Model on Hailo-8 – Wrong Bounding Boxes with Fused NMS Post-Processing

Setup

Issue
The .pt model in PyTorch detects objects fine.
But when I run the compiled .hef on Hailo-8, the bounding boxes don’t align with the objects. For example, detections land on unrelated regions (e.g. a person’s face) instead of the target object.

Example fused NMS output from Hailo:

yolov5s/yolov5_nms_postprocess (1, 1, 1, 5)
Raw dets: [[293.47 182.89 538.62 476.25]]

The coordinates look plausible, but they don’t match the object in the image.

My current post-processing functions:

def parse_hailo_nms(raw, img_hw):
    H, W = img_hw
    while isinstance(raw, list):
        raw = raw[0] if len(raw) else np.zeros((0, 5), dtype=np.float32)
    dets = np.asarray(raw).squeeze()
    if dets.size == 0:
        return np.zeros((0,4)), np.zeros((0,)), np.zeros((0,))
    if dets.ndim == 1:
        dets = dets.reshape(1, -1)

    N, C = dets.shape
    if C == 5:
        boxes = dets[:, 0:4].astype(np.float32)
        scores = dets[:, 4].astype(np.float32)
        clsids = np.zeros((N,), dtype=np.int32)
    else:
        boxes = dets[:, 0:4].astype(np.float32)
        scores = dets[:, 4].astype(np.float32)
        clsids = dets[:, 5].astype(np.int32)

    # scale boxes if normalized
    if boxes.max() <= 1.5:
        boxes[:, [0, 2]] *= W
        boxes[:, [1, 3]] *= H
    return boxes, scores, clsids

def undo_letterbox(boxes, r, dw, dh):
    if len(boxes) == 0: 
        return boxes
    boxes = boxes.copy()
    boxes[:, [0, 2]] = (boxes[:, [0, 2]] - dw) / r
    boxes[:, [1, 3]] = (boxes[:, [1, 3]] - dh) / r
    return boxes

My suspicion

  • I may be misinterpreting the fused NMS output tensor from Hailo.

  • Are the YOLOv5s fused NMS outputs already in input size (640×640) coordinates, or still normalized 0–1?

  • Am I applying undo_letterbox incorrectly?


What I need help with

  1. Correct interpretation of yolov5s/yolov5_nms_postprocess output format.

  2. Whether additional scaling/transform is required before mapping back to the original image.

  3. Example of a minimal correct post-processing pipeline for YOLOv5s fused NMS on Hailo.


:backhand_index_pointing_right: Has anyone already solved this mismatch for YOLOv5s on Hailo? Any reference code for mapping boxes back correctly would be super helpful :folded_hands:

Hi @Felix_Rottmann

Hailo output bbox format is yxyx and not xyxy. Can you please try swapping the coords and see if the new bbox coords make sense?

@shashi Thank you very much!!!
This was the mistake. Now the bounding box is sticking to the object tight and expected :sweat_smile:
You saved my Sunday - Thanks!

1 Like

@Felix_Rottmann

Glad to be of help. My one act of kindness for the sunday :slight_smile:

2 Likes