cascaded network failing

Hey everyone. I’m running a gstreamer pipeline below to try out some cascaded network architectures. I’m trying to detect an object, then crop and detect with another net. I have the pipeline running, and the detection network works great - however, I have no idea if the second network is doing anything at all. The only indication I have of any activity at all is that every so often, the gstreamer app will crash with the error:

Hailo Cropper Input and output caps have different formats

This error only seems to occur while I am getting a detection from the first network - and I suspect it happens only when the second network acquires a detection as well. However, I’m not sure how to resolve the issue, or where to even look.

Dot file

Pipeline:

hailomuxer name=hmux libcamerasrc name=src_0 ! tee name=vehicle_t ! queue name=process_bypass leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hmux.sink_0 vehicle_t. ! queue name=vehicle_in leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoscale ! videoconvert n-threads=3 !hailonet hef-path=/home/furysama/src/hailo-rpi5-examples/resources/models/hailo8l/yolov5m_vehicles_8l.hef device-count=1 is-active=true force-writable=true batch-size=2 vdevice-group-id=1 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 ! queue name=vehicle_post leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/libyolo_hailortpp_post.so function-name=yolov5m_vehicles qos=false ! queue name=vehicle_out leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hmux.sink_1 hmux. !queue name=vehicle_cropper_input_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailocropper name=vehicle_cropper_cropper so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/cropping_algorithms/liblpr_croppers.so function-name=vehicles_without_ocr use-letterbox=true no-scaling-bbox=true internal-offset=true resize-method=bilinear hailoaggregator name=vehicle_cropper_agg vehicle_cropper_cropper. ! queue name=vehicle_cropper_bypass_q leaky=no max-size-buffers=20 max-size-bytes=0 max-size-time=0 ! vehicle_cropper_agg.sink_0 vehicle_cropper_cropper. ! queue name=PLATE_DETECT_NET leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailonet hef-path=/home/furysama/src/hailo-rpi5-examples/resources/models/hailo8l/tiny_yolov4_license_plates.hef device-count=1 is-active=true force-writable=true batch-size=2 vdevice-group-id=1 !queue name=PLATE_DETECT_POST leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/libyolo_post.so function-name=tiny_yolov4_license_plates !queue name=PLATE_DETECT_OUT leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! vehicle_cropper_agg.sink_1 vehicle_cropper_agg. ! queue name=vehicle_cropper_output_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! hailooverlay line-thickness=3 font-thickness=1 qos=false ! identity name=identity_callback ! queue name=overlay leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoconvert ! autovideosink name=hailo_display sync=True

Hey @mike_f,

So the issue you’re running into is basically a format mismatch - the cropper is outputting in one format, but your second network is expecting something different.

What I’d suggest is adding a videoconvert element right after the cropper to handle the format conversion. Depending on your setup, you might also need to throw in a videoscale and maybe a capsfilter to make sure everything lines up properly between the two.

If you want to dig deeper into what’s actually failing, try running with GST_DEBUG=3 (or even higher if you need more detail). The verbose GStreamer logs should help you see exactly where the caps negotiation is breaking down.

Also, we actually have an old example of doing something similar in tappas that might be helpful - check it out here: GitHub - hailo-ai/tappas at v3.28.1

Hope that helps!

That makes sense - I’m currently using the python helper pipeline CROPPER_PIPELINE to build the cropper here - in this case, I’d need to build the whole thing myself, right?

I’ve updated my pipeline to look like below:
hailomuxer name=hmux libcamerasrc name=src_0 ! tee name=vehicle_t ! queue name=process_bypass leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hmux.sink_0 vehicle_t. ! queue name=vehicle_in leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! videoscale ! videoconvert n-threads=3 !hailonet hef-path=/home/furysama/src/hailo-rpi5-examples/resources/models/hailo8l/yolov5m_vehicles_8l.hef device-count=1 is-active=true force-writable=true batch-size=2 vdevice-group-id=1 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 ! queue name=vehicle_post leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/libyolo_hailortpp_post.so function-name=yolov5m_vehicles qos=false ! queue name=vehicle_out leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hmux.sink_1 hmux. !queue name=PLATE_DETECT_IN leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailocropper so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/cropping_algorithms/liblpr_croppers.so function-name=vehicles_without_ocr internal-offset=False no-scaling-bbox=TRUE use-letterbox=TRUE name=vehicle_cropper hailoaggregator name=vehicle_agg vehicle_cropper. ! video/x-raw, format=NV12 ! queue name=vehicle_cropper_bypass leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! vehicle_agg.sink_0 vehicle_cropper. ! video/x-raw,height=416,width=208,format=NV12 ! videoscale ! videoconvert n-threads=2 ! queue name=PLATE_DETECT_NET leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailonet hef-path=/home/furysama/src/hailo-rpi5-examples/resources/models/hailo8/tiny_yolov4_license_plates_nv12.hef device-count=1 is-active=true force-writable=true batch-size=2 vdevice-group-id=1 !queue name=PLATE_DETECT_POST leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! hailofilter so-path=/lib/aarch64-linux-gnu/hailo/tappas/post_processes/libyolo_post.so function-name=tiny_yolov4_license_plates config-path=/home/furysama/src/hailo-rpi5-examples/resources/json/yolov4_license_plate.json !queue name=PLATE_DETECT_OUT leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! vehicle_agg.sink_1 vehicle_agg. ! queue name=PLATE_CROP_OUT leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0 ! videoconvert ! hailooverlay line-thickness=3 font-thickness=1 qos=false ! queue name=overlay leaky=no max-size-buffers=30 max-size-bytes=0 max-size-time=0 ! autovideosink name=hailo_display sync=True

However, now I’m seeing:

xvimagebufferpool0 differen size, stride and/or offset, update
Segmentation fault

I’m not sure where this is being thrown - if its in the net, or post processing, or image sink. Maybe I’m still missing a videoconvert somewhere?