Hello, my board is rapi5 with Hailo-8L, I want to run with python “Hailo-Application-Code-Examples” from github, but I don’t know how to install Hailo Model Zoo
Welcome to the Hailo Community!
You do not need to install the Hailo Model Zoo. You can download precompiled models (HEFs) from the Model Zoo GitHub page. For instance Object detection models for Hailo-8L.
GitHub - Hailo Model Zoo - Hailo-8L - Object detection
To convert your own models you will need to install the Hailo AI Software Suite docker on a x86 Ubuntu machine. The suite comes with built in tutorials that show you the conversion process step by step. Just call the following command inside the docker.
hailo tutotorial
This will start a Jupyter Notebook server with notebooks for each step.
The suite also includes a hailomz command line tool that lets you convert models from the Model Zoo.
After going trough the tutorial call hailomz --help and you will notice the same steps (parse, optimize, compile) as in the general workflow. This will allow you to convert the models from the Model Zoo while using fewer parameters. The hailo_model_zoo directory contains yaml and alls files that tell hailomz what parameters to use to convert the models.
Hi
as I want to run this code in RPI5 with hailo8L (for fast_sam), but it requires hailo model zoo to be installed…how to proceed then?
https://github.com/hailo-ai/Hailo-Application-Code-Examples/blob/main/runtime/python/instance_segmentation/yoloseg_inference.py
To get started, please visit our Hailo Model Zoo repository on GitHub:
You’ll find a Quick Start guide there that should walk you through the process. if you run into any issues or have questions, feel free to post them here in the community.
hi Klausk
thanks for your help,
When I run Hailo-Application-Code-Examples/runtime/python/streaming at main · hailo-ai/Hailo-Application-Code-Examples · GitHub,
The script will call Hailo model zoo but Raspi5 can’t install DFC then it also can’t install Hailo model zoo. Have you successfully run this on a Raspberry Pi 5?
Best regards
Sam
Ｈi Klausk sir,
Thanks for your help. I have followed your instruction to download model from git hub but When I run the “/Hailo-Application-Code-Examples/runtime/python/streaming/yolox_stream_inference.py”. it is same error as photo
facing same issue…any solution?
Hey
In this part of the example, we’re utilizing the Hailo Model Zoo for post-processing. Here’s the relevant code:
# Create a dictionary that maps tensor shapes to layer names (needed for post-processing)
layer_from_shape: dict = {infer_results[key].shape: key for key in infer_results.keys()}
from hailo_model_zoo.core.postprocessing.detection import yolo
# Post-processing configuration, as recommended in hailo_model_zoo/cfg/base/yolox.yaml
anchors = {"strides": [32, 16, 8], "sizes": [[1, 1], [1, 1], [1, 1]]}
yolox_post_proc = yolo.YoloPostProc(
img_dims=(INPUT_RES_H, INPUT_RES_W),
nms_iou_thresh=0.65,
score_threshold=0.01,
anchors=anchors,
output_scheme=None,
classes=80,
labels_offset=1,
meta_arch="yolox",
device_pre_post_layers=[]
)
# The order here is crucial since the reorganized tensor must be in (BS,H,W,85) shape
endnodes = [
infer_results[layer_from_shape[1, 80, 80, 4]], # stride 8
infer_results[layer_from_shape[1, 80, 80, 1]], # stride 8
infer_results[layer_from_shape[1, 80, 80, 80]], # stride 8
infer_results[layer_from_shape[1, 40, 40, 4]], # stride 16
infer_results[layer_from_shape[1, 40, 40, 1]], # stride 16
infer_results[layer_from_shape[1, 40, 40, 80]], # stride 16
infer_results[layer_from_shape[1, 20, 20, 4]], # stride 32
infer_results[layer_from_shape[1, 20, 20, 1]], # stride 32
infer_results[layer_from_shape[1, 20, 20, 80]] # stride 32
]
# Process the outputs
hailo_preds = yolox_post_proc.yolo_postprocessing(endnodes)
num_detections = int(hailo_preds['num_detections'])
scores = hailo_preds["detection_scores"][0].numpy()
classes = hailo_preds["detection_classes"][0].numpy()
boxes = hailo_preds["detection_boxes"][0].numpy()
# Set the number of detections to 0 if the first score is 0
if scores[0] == 0:
num_detections = 0
# Create a dictionary to hold prediction results
preds_dict = {
'scores': scores,
'classes': classes,
'boxes': boxes,
'num_detections': num_detections
}
# Report and visualize detections
frame = report.report_detections(
preds_dict,
frame,
scale_factor_x=orig_w,
scale_factor_y=orig_h
)
cv2.imshow('frame', frame)
If you’d like, you can either replace the operations performed by the Hailo Model Zoo with your own post-processing steps, or try running other examples. However, please note that these examples have primarily been tested on x86 architectures and are not optimized for Raspberry Pi (RPI) at this time. We’re working on versions specifically optimized for RPI, similar to what we did with the TAPPAS , which now works on RPI as RPI-examples.
Let me know if you encounter any issues or if you would like further assistance with this setup.
Best regards,
Omri
Hi
we cannot install hailo_model_zoo in rpi…so how to test the examples from Hailo-Application-Code-Examples in rpi ?
Specifically I want to test fast sam
Thank you
Hey @dario.ravarro,
If you take a look at this inference example: Hailo YOLOSeg Inference Example, you’ll notice that the post-processing function imports from the Hailo model zoo.
To use this example on the Raspberry Pi (RPI), you’ll need to write your own post-processing function and a function to visualize the results. That’s why we didn’t recommend using these examples directly until they are optimized. The best way to proceed would be to review the post-processing functions in the Hailo model zoo and integrate them into your example instead of importing them.
Here are the functions you’ll need:
def visualize_yolov5_seg_results(
detections, img, class_names=None, alpha=0.5, score_thres=0.25, mask_thresh=0.5, max_boxes_to_draw=20, **kwargs
):
img_idx = 0
img_out = img[img_idx].copy()
boxes = detections["detection_boxes"]
# Scale the boxes to match the input image dimensions
boxes[:, 0::2] *= img_out.shape[1]
boxes[:, 1::2] *= img_out.shape[0]
masks = detections["mask"] > mask_thresh
scores = detections["detection_scores"]
classes = detections["detection_classes"]
skip_boxes = kwargs.get("meta_arch", "") == "yolov8_seg_postprocess" and kwargs.get("classes", "") == 1
keep = scores > score_thres
boxes, masks, scores, classes = boxes[keep], masks[keep], scores[keep], classes[keep]
max_boxes = min(max_boxes_to_draw, len(keep))
boxes, masks, scores, classes = boxes[:max_boxes], masks[:max_boxes], scores[:max_boxes], classes[:max_boxes]
for idx, mask in enumerate(masks):
xmin, ymin, xmax, ymax = boxes[idx].astype(np.int32)
color = np.random.randint(0, 255, size=3, dtype=np.uint8)
if not skip_boxes:
img_out = cv2.rectangle(img_out, (xmin, ymin), (xmax, ymax), [int(c) for c in color], 3)
if np.sum(mask) > 0:
polygons, _ = mask_to_polygons(mask)
mask_overlay = np.repeat(mask[:, :, np.newaxis], 3, axis=2) * color
img_out = cv2.addWeighted(mask_overlay, alpha, img_out, 1, 0)
for polygon in polygons:
img_out = cv2.polylines(
img_out, [polygon.reshape((-1, 1, 2)).astype(np.int32)], isClosed=True, color=color, thickness=1
)
if not skip_boxes:
label = f"{class_names[int(classes[idx])]}"
score = f"{int(100 * scores[idx])}%"
text = f"{label}: {score}"
(w, h), _ = cv2.getTextSize(text, fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=0.5, thickness=2)
org = (xmin, ymin)
img_out = cv2.putText(img_out, text, org, fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=0.5, color=[255, 255, 255], thickness=2)
return img_out
def yolov8_seg_postprocess(endnodes, device_pre_post_layers=None, **kwargs):
num_classes = kwargs["classes"]
strides = kwargs["anchors"]["strides"][::-1]
image_dims = tuple(kwargs["img_dims"])
reg_max = kwargs["anchors"]["regression_length"]
raw_boxes = endnodes[:7:3]
scores = [np.reshape(s, (-1, s.shape[1] * s.shape[2], num_classes)) for s in endnodes[1:8:3]]
scores = np.concatenate(scores, axis=1)
decoded_boxes = _yolov8_decoding(raw_boxes, strides, image_dims, reg_max)
score_thres = kwargs["score_threshold"]
iou_thres = kwargs["nms_iou_thresh"]
proto_data = endnodes[9]
batch_size, _, _, n_masks = proto_data.shape
fake_objectness = np.ones((scores.shape[0], scores.shape[1], 1))
scores_obj = np.concatenate([fake_objectness, scores], axis=-1)
coeffs = [np.reshape(c, (-1, c.shape[1] * c.shape[2], n_masks)) for c in endnodes[2:9:3]]
coeffs = np.concatenate(coeffs, axis=1)
predictions = np.concatenate([decoded_boxes, scores_obj, coeffs], axis=2)
nms_res = non_max_suppression(predictions, conf_thres=score_thres, iou_thres=iou_thres, multi_label=True)
outputs = []
for b in range(batch_size):
protos = proto_data[b]
masks = process_mask(protos, nms_res[b]["mask"], nms_res[b]["detection_boxes"], image_dims, upsample=True)
output = {
"detection_boxes": np.array(nms_res[b]["detection_boxes"]),
"mask": np.transpose(masks, (0, 1, 2)) if masks is not None else None,
"detection_scores": np.array(nms_res[b]["detection_scores"]),
"detection_classes": np.array(nms_res[b]["detection_classes"]).astype(int)
}
outputs.append(output)
return outputs
Let me know if you have any questions!
Regards
I copied basically most of the content of instance_segmentation_postprocessing.py into yoloseg_inference and also installed cython to be able to import cython_nms.pyx.
Where can I upload the yoloseg_inference.py?
Running
python yoloseg_inference.py fast_sam_s.hef dog_bicycle.jpg fast
generates an output image in a new folder, with segmentation.
However the segmentation is missing some parts… How to fine tune it?
How to run segmentation on live stream from rpi camera?
1 Like
Great job!
It seems the missing segmentation might be related to a post-processing issue. The object is being detected, but the coverage isn’t complete across the entire object. I recommend making some adjustments to the post-processing and testing it again. Since the app was initially developed for x86 (PC) or VMS with Hailo, it might require some modifications in post-processing to work properly in this setup.
As for live streaming, you can refer to our implementation here: Hailo Live Stream Example.
The live stream example doesn’t work with RPI camera.
I am trying to use rpicamera2 in the segmentation code, but it’s unclear how to implement it properly in yoloseg_inference.py
how to modify yoloseg_inference to get images from RPI camera?
@omria I’m inferencing yolov8 instance segmentation model and depth model parallel on the device Hailo-8, I manually modified from (hailo_models_zoo) post-processing functions for the segmentation model , But the issue is that the post-processing function takes a longer time than the model inference time, I see there is potential solutions using libyolo_post.so functions available on Tappas, using GStreamer can significantly reduce post-processing time , but I planing use multiprocessing instead of GStreamer, is there possible suggestions using libyolo_post function without gstreamer?