Tracking messes up instance segmentation in example pipeline app

General Discussion
1

I am running into an issue running the python instance_segmentation pipeline app example. The segmentation masks do not match the image very well.

I was saving the frames of my custom app for review, and I noticed that the segmentation masks did not match the images. After spending a while trying to figure out what went wrong, I tried the python example app and noticed the same issues.

At this point, I realized that the last time I had seen good segmentation masks was before I added tracking to my app, so I removed it from the example’s instance_segmentation_pipeline.py file. With that removed, the segmentation masks align as expected.

I made a small edit to the example code to output a few frames for review:
Before edit to instance_segmentation_pipeline.py:

frame_2: see feet of man in foreground (cyan) and legs of man on the right (green)
frame_2: see feet of man in foreground (cyan) and legs of man on the right (green)640×640 114 KB

After edit to line 171 of instance_segmentation_pipeline.py:

        pipeline_string = (
            f"{source_pipeline} ! "
            f"{infer_pipeline_wrapper} ! "
            # f"{tracker_pipeline} ! "
            f"{user_callback_pipeline} ! "
            f"{display_pipeline}"
        )

frame_2
frame_2640×640 112 KB

(note, the lack of different colors is due to all detection’s having the same id due to the removed tracking)

In some frames the mask looks like it was held from a prior frame. In others the mask doesn’t fit any surrounding frame.

Just to make sure I didn’t break anything, I did a “git restore” & “git cleanup” on the hailo-apps directory to remove any changes that may be causing it. I still get the same results.

At one point I thought it might be an issue with the default yolov5m_seg model, so I tried the yolov8m_seg. It took a bit to get it working, since the postprocessing c++ code for that model didn’t exists, but once I did I saw the same behavior.

Can anyone else confirm if this behavior?

I am using an RPi5 with a relatively fresh install of Raspberry PI OS (62-bit), with a Hailo8 AI+ hat.

$ hostnamectl
 Static hostname: xxxxxxxxxxxxxxx
       Icon name: xxxxxxxxxxxxxxx
      Machine ID: xxxxxxxxxxxxxxx
         Boot ID: xxxxxxxxxxxxxxx
Operating System: Debian GNU/Linux 13 (trixie)    
          Kernel: Linux 6.12.75+rpt-rpi-2712
    Architecture: arm64


$ hailortcli fw-control identify
Executing on device: 0001:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.23.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8
2

Hi @Mark_Robertson,

The HailoTracker uses a Kalman filter to keep objects alive through brief detection misses (if they occur).

The tracker creates a predicted bounding box and carries forward all the old metadata unchanged - including the segmentation mask from the last real detection. So you get a stale pixel-level mask from frame N shown on frame N+i at a new predicted box position.

Thanks,