Yolo segmentation converted to hef file dont work

Dear community,
I think that hailo has work more in order to allow developers to use more easily the promising systems that they are providing.

Anyway i have followed this tutorial Raspberry Pi AI Kit: ONNX to HEF Conversion and i have done the translation from onnx yolo segment to the hef file by using the dataflow compiler in windows.

when i try to run the follow command in order to test the yolo segmentation model in local mp4

(venv_hailo_rpi5_examples) pi@raspberrypi:~/hailo-rpi5-examples $ python3 basic_pipelines/detection.py --hef-path /home/pi/Desktop/segmentation/yolov8n_seg.hef --input /home/pi/Desktop/segmentation/Progetto_video.mp4

this what i have seen in the output

Auto-detected Hailo architecture: hailo8l
filesrc location="/home/pi/Desktop/segmentation/Progetto_video.mp4" name=source ! queue name=source_queue_dec264 leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! qtdemux ! h264parse ! avdec_h264 max-threads=2 !  queue name=source_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=source_videoscale n-threads=2 ! queue name=source_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert n-threads=3 name=source_convert qos=false ! video/x-raw, format=RGB, pixel-aspect-ratio=1/1 !  queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=inference_videoscale n-threads=2 qos=false ! queue name=inference_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! video/x-raw, pixel-aspect-ratio=1/1 ! videoconvert name=inference_videoconvert n-threads=2 ! queue name=inference_hailonet_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailonet name=inference_hailonet hef-path=/home/pi/Desktop/segmentation/yolov8n_seg.hef batch-size=2 nms-score-threshold=0.3 nms-iou-threshold=0.45 output-format-type=HAILO_FORMAT_TYPE_FLOAT32 force-writable=true ! queue name=inference_hailofilter_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailofilter name=inference_hailofilter so-path=/home/pi/hailo-rpi5-examples/basic_pipelines/../resources/libyolo_hailortpp_postprocess.so   qos=false  ! queue name=identity_callback_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! identity name=identity_callback  ! queue name=hailo_display_hailooverlay_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailooverlay name=hailo_display_hailooverlay ! queue name=hailo_display_videoconvert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert name=hailo_display_videoconvert n-threads=2 qos=false ! queue name=hailo_display_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! fpsdisplaysink name=hailo_display video-sink=xvimagesink sync=true text-overlay=false signal-fps-measurements=true 
Showing FPS
NMS score threshold is set, but there is no NMS output in this model.
CHECK_SUCCESS failed with status=6

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.136: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.141: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.160: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.165: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.178: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.180: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.181: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.194: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.200: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.201: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.209: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.223: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.226: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.232: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.241: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.244: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.250: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.259: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.267: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.275: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.285: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.289: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.305: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.306: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.326: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.326: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.339: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.344: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.356: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.361: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.381: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.386: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.396: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.407: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.411: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.422: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.439: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.444: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.449: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.463: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.468: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.487: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.499: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.506: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.515: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.516: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.529: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.534: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.562: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.562: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.588: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.593: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.608: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.616: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.629: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.637: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.654: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.665: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.672: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.685: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.695: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.703: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.724: gst_queue_array_push_tail: assertion 'array != NULL' failed

(Hailo Detection App:6195): GStreamer-Base-CRITICAL **: 14:37:09.725: gst_queue_array_push_tail: assertion 'array != NULL' failed

someone can help me to understand the problem and how to fix it?

@agronomofiorentini
One issue i can see you’re running objection detection file not segmentation. you need to run this file hailo-rpi5-examples/basic_pipelines/instance_segmentation.py at main · hailo-ai/hailo-rpi5-examples · GitHub

1 Like

thanks to reply me.

i have changed the python script but i have got

(venv_hailo_rpi5_examples) pi@raspberrypi:~/hailo-rpi5-examples $ python3 basic_pipelines/instance_segmentation.py --hef-path /home/pi/Desktop/segmentation/yolov8n_seg.hef --input /home/pi/Desktop/segmentation/Progetto_video.mp4
Auto-detected Hailo architecture: hailo8l
Traceback (most recent call last):
  File "/home/pi/hailo-rpi5-examples/basic_pipelines/instance_segmentation.py", line 96, in <module>
    app = GStreamerInstanceSegmentationApp(app_callback, user_data)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/hailo-rpi5-examples/basic_pipelines/instance_segmentation_pipeline.py", line 68, in __init__
    raise ValueError("HEF version not supported, you will need to provide a config file")
ValueError: HEF version not supported, you will need to provide a config file

what should i do?

I think this is the issue you can add your model name. For simplicity rename your hef file accordingly. Or add your model name in code.

1 Like

i have changed from



    if 'yolov5m_seg' in self.hef_path:
        self.config_file = os.path.join(self.current_path, '../resources/yolov5m_seg.json')
    elif 'yolov5n_seg' in self.hef_path:
        self.config_file = os.path.join(self.current_path, '../resources/yolov5n_seg.json')
    else:
        raise ValueError("HEF version not supported, you will need to provide a config file")


to

        if 'cytron' in self.hef_path:
            self.config_file = os.path.join(self.current_path, '../resources/yolov5m_seg.json')
        elif 'cytron' in self.hef_path:
            self.config_file = os.path.join(self.current_path, '../resources/yolov5n_seg.json')
        else:
            raise ValueError("HEF version not supported, you will need to provide a config file")

and run again

(venv_hailo_rpi5_examples) pi@raspberrypi:~/hailo-rpi5-examples $ python3 basic_pipelines/instance_segmentation.py --hef-path resources/cytron.hef --input /home/pi/Desktop/segmentation/Progetto_video.mp4
Auto-detected Hailo architecture: hailo8l
filesrc location="/home/pi/Desktop/segmentation/Progetto_video.mp4" name=source ! queue name=source_queue_dec264 leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! qtdemux ! h264parse ! avdec_h264 max-threads=2 !  queue name=source_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=source_videoscale n-threads=2 ! queue name=source_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert n-threads=3 name=source_convert qos=false ! video/x-raw, format=RGB, pixel-aspect-ratio=1/1 !  queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=inference_videoscale n-threads=2 qos=false ! queue name=inference_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! video/x-raw, pixel-aspect-ratio=1/1 ! videoconvert name=inference_videoconvert n-threads=2 ! queue name=inference_hailonet_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailonet name=inference_hailonet hef-path=resources/cytron.hef batch-size=2  force-writable=true ! queue name=inference_hailofilter_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailofilter name=inference_hailofilter so-path=/home/pi/hailo-rpi5-examples/basic_pipelines/../resources/libyolov5seg_postprocess.so  config-path=/home/pi/hailo-rpi5-examples/basic_pipelines/../resources/yolov5m_seg.json   function-name=yolov5seg  qos=false  ! queue name=identity_callback_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! identity name=identity_callback  ! queue name=hailo_display_hailooverlay_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailooverlay name=hailo_display_hailooverlay ! queue name=hailo_display_videoconvert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert name=hailo_display_videoconvert n-threads=2 qos=false ! queue name=hailo_display_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! fpsdisplaysink name=hailo_display video-sink=xvimagesink sync=true text-overlay=false signal-fps-measurements=true 
Showing FPS
Segmentation fault

Which yolo model(n/m) you are using ? Based on that you need to rename.

In case you’re using v8n postprocessing is different and for v8m is different. So please make sure.

1 Like

i am using yolov8n-seg.

What i should do?

Only change elif line…
Currently as i can see (m) model postprocessing is being used.

1 Like

Thanks to help me with the issue.

But i am lost

this what i have done

        # self.default_post_process_so = os.path.join(self.postprocess_dir, 'libyolov5seg_post.so')
        if 'yolov5m_seg' in self.hef_path:
            self.config_file = os.path.join(self.current_path, '../resources/yolov5m_seg.json')
        elif 'cytron' in self.hef_path:
            self.config_file = os.path.join(self.current_path, '../resources/cytron-labels.json')
        #else:
            #raise ValueError("HEF version not supported, you will need to provide a config file")
        self.default_post_process_so = os.path.join(self.current_path, '../resources/libyolov5seg_postprocess.so')
        self.post_function_name = "yolov5seg"
        self.app_callback = app_callback

this is my cytron-labels.json

{
    "iou_threshold": 0.1,
    "detection_threshold": 0.7,
    "output_activation": "none",
    "label_offset":1,
    "max_boxes":200,
    "anchors": [
     [ 116, 90, 156, 198, 373, 326 ],
     [ 30, 61, 62, 45, 59, 119 ],
     [ 10, 13, 16, 30, 33, 23 ]
    ],
    "labels": [
     "Grape",
     "Trunk",
     "Cordon",
     "Shoot",
     "Canopy"       
    ]
   }

this is the cmd lunched and related error

(venv_hailo_rpi5_examples) pi@raspberrypi:~/hailo-rpi5-examples $ python3 basic_pipelines/instance_segmentation.py --hef-path resources/cytron.hef --input /home/pi/Desktop/segmentation/Progetto_video.mp4
Auto-detected Hailo architecture: hailo8l
filesrc location="/home/pi/Desktop/segmentation/Progetto_video.mp4" name=source ! queue name=source_queue_dec264 leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! qtdemux ! h264parse ! avdec_h264 max-threads=2 !  queue name=source_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=source_videoscale n-threads=2 ! queue name=source_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert n-threads=3 name=source_convert qos=false ! video/x-raw, format=RGB, pixel-aspect-ratio=1/1 !  queue name=inference_scale_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoscale name=inference_videoscale n-threads=2 qos=false ! queue name=inference_convert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! video/x-raw, pixel-aspect-ratio=1/1 ! videoconvert name=inference_videoconvert n-threads=2 ! queue name=inference_hailonet_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailonet name=inference_hailonet hef-path=resources/cytron.hef batch-size=2  force-writable=true ! queue name=inference_hailofilter_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailofilter name=inference_hailofilter so-path=/home/pi/hailo-rpi5-examples/basic_pipelines/../resources/libyolov5seg_postprocess.so  config-path=/home/pi/hailo-rpi5-examples/basic_pipelines/../resources/cytron-labels.json   function-name=yolov5seg  qos=false  ! queue name=identity_callback_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! identity name=identity_callback  ! queue name=hailo_display_hailooverlay_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! hailooverlay name=hailo_display_hailooverlay ! queue name=hailo_display_videoconvert_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! videoconvert name=hailo_display_videoconvert n-threads=2 qos=false ! queue name=hailo_display_q leaky=no max-size-buffers=3 max-size-bytes=0 max-size-time=0  ! fpsdisplaysink name=hailo_display video-sink=xvimagesink sync=true text-overlay=false signal-fps-measurements=true 
Showing FPS
Input JSON is invalid
Invalid schema: #
Invalid keyword: required
Invalid document: #
terminate called after throwing an instance of 'std::runtime_error'
  what():  json config file doesn't follow schema rules
Aborted

Please try first the default config for yolov8n seg. And if this works, copy and modify that file…

Sorry i little bit confusion about the model. You said you are using yolov8n but this is for yolov5 seg.

But I believe there is also a post processing for yolov8n seg…
I will update you.

I can not find postprocess library for yolov8 seg. But they have created example using python api and cpp. You can take a look. You will need to spend some more time understanding device initialization and configurations…

There is a Readme file in both examples dir. You can follow to download default model and image to test then you will have to update code to use a video.