I’m struggeling to get my ai pipeline to keep up with the live rtsp stream.
Once in a while a frame gets dropped, causing my ai-stream to become out of sync after running for a while. (i clamped the frame rate to 14)
fyi:
If I leave out the frame rate limiting, then the fps hits 15.3 (because othe streams are inspected as wel with the hailo chip) however in that case everything gets jammed up pretty soon in the ai rtsp feedback stream.
my python code is:
source = (
f"rtspsrc location=\\"{self.ai_source}\\" name={name} ! "
f"{QUEUE(name=f'{name}\_queue_decode')} ! "
f"decodebin name={name}\_decodebin ! "
f"{QUEUE(name=f'{name}\_scale_q')} ! "
f"videoscale name={name}\_videoscale n-threads=2 ! "
f"{QUEUE(name=f'{name}\_convert_q')} ! "
f"videoconvert n-threads=3 name={name}\_convert qos=false ! "
f"video/x-raw, pixel-aspect-ratio=1/1, format={video_format}, "
f"width={self.video_width}, height={self.video_height} ! "
f"videorate name={name}\_videorate ! "
f"capsfilter name={name}\_fps_caps caps=\\"{fps_caps}\\" "
)
inference = INFERENCE_PIPELINE(
hef_path=cfg\["hef_path"\],
config_json=cfg\["config_json"\],
post_process_so=cfg\["postprocess_so"\],
post_function_name=cfg\["post_function_name"\],
batch_size=int(settings.tiles\[0\]) \* int(settings.tiles\[1\]),
)
processing = TILE_CROPPER_PIPELINE(
inner_pipeline=inference,
tiling_mode=0,
tiles_along_x_axis=int(settings.tiles\[0\]),
tiles_along_y_axis=int(settings.tiles\[1\]),
overlap_x_axis=0.0,
overlap_y_axis=0.0,
iou_threshold=0.15,
border_threshold=0.1,
scale_level=0,
)
tracker = TRACKER_PIPELINE(
class_id=-1,
kalman_dist_thr=0.8,
iou_thr=0.9,
init_iou_thr=0.7,
keep_new_frames=2,
keep_tracked_frames=15,
keep_lost_frames=2,
keep_past_metadata=True,
qos=False,
name="hailo_tracker",
)
callback = USER_CALLBACK_PIPELINE()
ai_rtsp_stream = (
f"{QUEUE(name='rtsp_convert_q')} ! "
f"videoconvert n-threads=2 ! "
f"video/x-raw,format=I420 ! "
f"openh264enc complexity=low bitrate=2000000 ! "
f"video/x-h264,profile=baseline ! "
f"h264parse config-interval=-1 ! "
f"rtspclientsink location={self.ai_output} protocols=tcp latency=0"
)
return (
f"{source} ! "
f"{processing} ! "
f"{tracker} ! "
f"{callback} ! "
f"{ai_rtsp_stream}"
)
this results in the following pipeline:
rtspsrc location=“rtsp://localhost:8554/live/9” name=source ! queue name=source_queue_decode leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! decodebin name=source_decodebin ! queue name=source_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoscale name=source_videoscale n-threads=2 ! queue name=source_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoconvert n-threads=3 name=source_convert qos=false ! video/x-raw, pixel-aspect-ratio=1/1, format=RGB, width=1280, height=720 ! videorate name=source_videorate ! capsfilter name=source_fps_caps caps=“video/x-raw, framerate=14/1” ! queue name=tile_cropper_wrapper_input_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailotilecropper name=tile_cropper_wrapper_cropper internal-offset=true tiling-mode=0 tiles-along-x-axis=2 tiles-along-y-axis=1 overlap-x-axis=0.0 overlap-y-axis=0.0 hailotileaggregator name=tile_cropper_wrapper_agg flatten-detections=true iou-threshold=0.15 border-threshold=0.1 tile_cropper_wrapper_cropper. ! queue name=tile_cropper_wrapper_bypass_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! tile_cropper_wrapper_agg. tile_cropper_wrapper_cropper. ! queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoscale name=inference_videoscale n-threads=2 qos=false ! queue name=inference_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! video/x-raw, pixel-aspect-ratio=1/1 ! videoconvert name=inference_videoconvert n-threads=2 ! queue name=inference_hailonet_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailonet name=inference_hailonet hef-path=/home/conneqt/security-camera/models/yolo26s.hef batch-size=2 vdevice-group-id=SHARED force-writable=true ! queue name=inference_hailofilter_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailofilter name=inference_hailofilter so-path=/home/conneqt/security-camera/postprocess/libyolo26_postprocess.so config-path=/home/conneqt/security-camera/yolo_config26.json function-name=yolo26 qos=false ! queue name=inference_output_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! tile_cropper_wrapper_agg. tile_cropper_wrapper_agg. ! queue name=tile_cropper_wrapper_output_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailotracker name=hailo_tracker class-id=-1 kalman-dist-thr=0.8 iou-thr=0.9 init-iou-thr=0.7 keep-new-frames=2 keep-tracked-frames=15 keep-lost-frames=2 keep-past-metadata=True qos=False ! queue name=hailo_tracker_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! queue name=identity_callback_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! identity name=identity_callback ! queue name=rtsp_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoconvert n-threads=2 ! video/x-raw,format=I420 ! openh264enc complexity=low bitrate=2000000 ! video/x-h264,profile=baseline ! h264parse config-interval=-1 ! rtspclientsink location=rtsp://localhost:8554/ai/9 protocols=tcp latency=0
The setup is:
rtsp → inference → tiling (2x1) → tracking → python callback → ai rtsp restream
I’ve got the feeling i need to put “leaky” somewhere in the pipeline, but until now I had no luck with that. I’m new to gstreamer.
Concreate question:
What must i do to keep the ai in sync with the live stream without clamping the frame rate, taking as much frames per seconds as possible. Something like “allow_dropped_frames = True“