Comfiling Custom hef file for detection and segmentation using hailo_model_zoo

General
,
1

Hello Hailo!
i have modified to infer classification model below link and checked inference result \

https://github.com/hailo-ai/Hailo-Application-Code-Examples/tree/main/runtime/python/instance_segmentation

Nowaday, I am trying to modify for instance segmentation and object detection.
this purpose of segmentation is that image save using custom hef file, with segment overlay data.
as another need for it, purpose of detection is that image save using custom hef file, with bounding overlay box data.

At the above link, source version required is hailort 4.16.0, and I am using hailort 4.17.0, I think that is able to modify.

then, I have several questions.

the resource code ‘yoloseg_inference.py’ and i am modifying by spliting into detection and segment.
when running ./yoloseg_inference.py as non-modified, it occured KeyError and I resolved by editting endnodes layers.

  • Segmentataion Customization
    I entered the output nodes into detection channels at postprc_yolov5 function.
    I changed mask_channels according to detection_channels
    As a result, It has been occured error as shown below:

ValueError Traceback (most recent call last)
Cell In [1], line 165
163 raw_detections = infer_pipeline.infer(input_data)
164 print(kwargs[‘meta_arch’])
→ 165 results = func_dictmeta_arch[0]
167 #results = detection_postprocessing(raw_detections[output_vstream_info.name], **kwargs)
169 output_path = os.path.join(os.path.realpath(‘.’), ‘output_images’)
Cell In [1], line 60, in postproc_yolov5det(raw_detections)
55 endnodes = [raw_detections[layer_from_shape[1,7, 7, 27]],
56 raw_detections[layer_from_shape[1,14, 14, 27]],
57 raw_detections[layer_from_shape[1,28, 28, 27]]]
58 #print(endnodes)
—> 60 predictions_dict = detection_postprocessing(endnodes, **kwargs)
62 return predictions_dict
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/detection_postprocessing.py:45, in detection_postprocessing(endnodes, device_pre_post_layers, **kwargs)
43 kwargs[“device_pre_post_layers”] = device_pre_post_layers
44 postproc = _get_postprocessing_class(meta_arch)(**kwargs)
—> 45 return postproc.postprocessing(endnodes, **kwargs)
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/detection/yolo.py:323, in YoloPostProc.postprocessing(self, endnodes, **kwargs)
321 return self.iou_nms(detection_boxes, detection_scores)
322 else:
→ 323 return self.yolo_postprocessing(endnodes, **kwargs)
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/detection/yolo.py:123, in YoloPostProc.yolo_postprocessing(self, endnodes, **kwargs)
121 anchors_for_stride = np.reshape(anchors_for_stride, (1, 1, -1, 2)) # dim [1, 1, 3, 2]
122 output_branch_and_data = [output_branch, anchors_for_stride, stride]
→ 123 detection_boxes, detection_scores = tf.numpy_function(self.yolo_postprocess_numpy,
124 output_branch_and_data,
125 [‘float32’, ‘float32’],
126 name=f’{self._network_arch}_postprocessing’)
128 # detection_boxes is a [BS, num_detections, 1, 4] tensor, detection_scores is a
129 # [BS, num_detections, num_classes] tensor
130 BS = endnodes[0].shape[0]
File ~/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback..error_handler(*args, **kwargs)
151 except Exception as e:
152 filtered_tb = _process_traceback_frames(e.traceback)
→ 153 raise e.with_traceback(filtered_tb) from None
154 finally:
155 del filtered_tb
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/detection/yolo.py:191, in YoloPostProc.yolo_postprocess_numpy(self, net_out, anchors_for_stride, stride)
188 offsets = np.expand_dims(np.expand_dims(offsets, 0), 0) # dim [1,1,128,128,2]
190 pred = net_out.transpose((0, 3, 1, 2)) # now dims are: [N,C,H,W] as in Gluon.
→ 191 pred = np.reshape(pred, (BS, num_anchors * num_pred, -1)) # dim [N, 255, HxW]
192 # dim [N, 361, 255], we did it so that the 255 be the last dim and can be reshaped.
193 pred = pred.transpose((0, 2, 1))
File <array_function internals>:180, in reshape(*args, **kwargs)
File ~/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py:298, in reshape(a, newshape, order)
198 @array_function_dispatch(_reshape_dispatcher)
199 def reshape(a, newshape, order=‘C’):
200 “”"
201 Gives a new shape to an array without changing its data.
202
(…)
296 [5, 6]])
297 “”"
→ 298 return _wrapfunc(a, ‘reshape’, newshape, order=order)
File ~/hailo_ai_sw_suite/hailo_venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py:57, in _wrapfunc(obj, method, *args, **kwds)
54 return _wrapit(obj, method, *args, **kwds)
56 try:
—> 57 return bound(*args, **kwds)
58 except TypeError:
59 # A TypeError occurs if the object does have such a method in its
60 # class, but its signature is not identical to that of NumPy’s. This
(…)
64 # Call _wrapit from within the except clause to ensure a potential
65 # exception has a traceback chain.
66 return _wrapit(obj, method, *args, **kwds)
ValueError: cannot reshape array of size 1323 into shape (1,30,newaxis)

  • Detection Code Customization
    For the detection process, I modified postprc_yolov5, that I entered detection_channels to output nodes.
    and It occured like below:

ValueError Traceback (most recent call last)
Cell In [1], line 168
165 with network_group.activate(network_group_params):
166 raw_detections = infer_pipeline.infer(input_data)
→ 168 results = func_dictmeta_arch[0]
170 output_path = os.path.join(os.path.realpath(‘.’), ‘output_images’)
171 if not os.path.isdir(output_path):
Cell In [1], line 51, in postproc_yolov5seg(raw_detections)
46 endnodes = [raw_detections[layer_from_shape[1, 28, 28, 120]],
47 raw_detections[layer_from_shape[1, 14, 14, 120]],
48 raw_detections[layer_from_shape[1, 7, 7, 120]]]
50 print(raw_detections_keys)
—> 51 predictions_dict = yolov5_seg_postprocess(endnodes, **kwargs)
53 return predictions_dict
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/instance_segmentation_postprocessing.py:541, in yolov5_seg_postprocess(endnodes, device_pre_post_layers, **kwargs)
539 iou_thres = kwargs[‘nms_iou_thresh’]
540 outputs = non_max_suppression(outputs, score_thres, iou_thres, nm=protos.shape[-1])
→ 541 outputs = _finalize_detections_yolov5_seg(outputs, protos, **kwargs)
543 # reorder and normalize bboxes
544 for output in outputs:
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/instance_segmentation_postprocessing.py:581, in _finalize_detections_yolov5_seg(outputs, protos, **kwargs)
578 masks = output[‘mask’]
579 proto = protos[batch_idx]
→ 581 masks = process_mask(proto, masks, boxes, shape, upsample=True)
582 output[‘mask’] = masks
584 return outputs
File ~/hailo_ai_sw_suite/sources/model_zoo/hailo_model_zoo/core/postprocessing/instance_segmentation_postprocessing.py:452, in process_mask(protos, masks_in, bboxes, shape, upsample, downsample)
450 mh, mw, c = protos.shape
451 ih, iw = shape
→ 452 masks = _sigmoid(masks_in @ protos.reshape((-1, c)).transpose((1, 0))).reshape((-1, mh, mw))
454 downsampled_bboxes = bboxes.copy()
455 if downsample:
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 120 is different from 32)

I didn’t modify yoloseg_inference.py other than preproc_yolov5 function, I removed only an line into first of endenodes as show below:
raw_detections[layer_from_shape[1, 160, 160, mask_channels]],

Could you please let me know what is values of width and height into the layer_from_shape?

and also please let me know how to get rid of traceback error.

def postproc_yolov5seg(raw_detections):

raw_detections_keys = list(raw_detections.keys())
layer_from_shape: dict = {raw_detections[key].shape:key for key in raw_detections_keys}

#mask_channels = 31

#detection_channels = (kwargs['classes'] + 4 + 1 + mask_channels) *  len(kwargs['anchors']['strides']) # (num_classes + num_coordinates + objectness + mask) * strides_list_len 
#print(len(kwargs['anchors']['strides']))
endnodes = [raw_detections[layer_from_shape[1, 28, 28, 120]],
            raw_detections[layer_from_shape[1, 14, 14, 120]],
            raw_detections[layer_from_shape[1, 7, 7, 120]]]

print(raw_detections_keys)
predictions_dict = yolov5_seg_postprocess(endnodes, **kwargs)

return predictions_dict
  • Additional info
  • I am using hailo_model_zoo API

thanks :slight_smile:

2

I knew that it required mask protos value on the postproc function.

I found mask informations at netron and

i found I have two outputs into segmentation model through Netron website.

so, when i parsed to onnx model at dfc1_parsing_example,

I have been entered two endnodes name into translate function.

(before write topic, I entered only one output.)

and, I exported to hef file that applied two output node names.

then, the newly generated hef file works on my coustom segmentation model.

Finally, After resolved, I provide the method I discovered.