Is there any oriented bounding box model object detection

SAN · September 25, 2024, 3:57am

im actually working on oriented bounding box instead of axis aligned bounding box is there any model developed on hailo model zoo

omria · September 25, 2024, 1:09pm

Hey @SAN

Our Model Zoo doesn’t currently include models specifically designed for OBB detection out of the box. Most of our object detection models, including our YOLO variants, use Axis-Aligned Bounding Boxes (AABB).

However, it’s entirely possible to adapt existing models like YOLOv5 or YOLOv8 to support OBB. Here’s a brief overview of how you could approach this:

Model Adjustments: Modify the final layer of the detection model to output 5 parameters: (x_center, y_center, width, height, angle). The angle will define the rotation of the bounding box.
Post-Processing: Implement Rotated Non-Maximum Suppression (Rotated NMS) for post-processing. This will handle the angle as part of the box definition.

You can still use our Hailo SDK and Dataflow Compiler for quantization and deployment on Hailo hardware. The custom OBB handling can be integrated using HailoRT-pp or through direct modifications to the neural network output.

If you need more detailed guidance on modifying a specific model from our Model Zoo to support OBB, please let me know. I’d be happy to provide more in-depth assistance.

SAN · September 26, 2024, 1:32am

thank for the information. i have actually done the model adjustment part, right now during the onnx translation after some research what i did was get the head layer of my custom model and go for the further process, right now the problem is i need some guidance on the hailoRT-pp and the quantization process. in that case can you provide me the yolov8n model building process or the PP code. Thanks for the further response

SAN · September 26, 2024, 5:46am

Here is a detailed explanation on my custom model. im using a yolov8n-obb.pt model and as i found that hailo use the 6 convolution layers directly from the head so i have already output it as well as the other head that the obb layer are output. The thing is im having a trouble on how to postprocess this part. So if i can get the yolov8n hailort-pp step it will be usefull. Thanks

omria · September 29, 2024, 1:36am

Hey @SAN,

Great to hear that you’ve made progress with adjusting the YOLOv8n-OBB model!

For the post-processing (HailoRT-PP) and quantization steps:

Post-Processing for OBB:
Since you’re dealing with oriented bounding boxes (OBBs), you’ll need to implement Rotated Non-Maximum Suppression (Rotated NMS) in the post-processing step. The default HailoRT-PP for YOLO models works with Axis-Aligned Bounding Boxes (AABB), so this is where you’ll need customization. You can adapt the post-processing to handle the (x_center, y_center, width, height, angle) format you’re working with.

You can reference the standard YOLOv8 post-processing configuration in the Hailo Model Zoo and modify it to include the additional angle parameter for OBB handling. Customizing the NMS to take the rotation into account is key here.
Quantization:
For quantization, make sure to provide calibration images that match your OBB format during the compilation process. You can use HailoRT’s quantization options, similar to how it’s done for YOLOv8 models in the Hailo Model Zoo. Focus on ensuring that the qp_scale and qp_zp parameters are set correctly in the model’s YAML configuration.
Model Compilation:
If you’re using the Hailo Model Zoo, you can follow the standard YOLOv8 compilation process and adjust the output head to accommodate the OBB-specific parameters. Make sure to modify both the post-processing step (to handle the angle) and the quantization configurations.

Next Steps:

Adjust your post-processing logic to support Rotated NMS.
Compile the model with the Dataflow Compiler and ensure proper quantization.
Modify the YAML configuration to handle the new output format, including the angle for the OBBs.

Let me know if you need more detailed guidance on the configuration or if you’d like some example code to help with the post-processing changes!

Regards

SAN · October 7, 2024, 1:58am

Hi, just for clarification, for yolov8n hailo used 6 end_node before the detect layer model22 then postprocess the output. FOR yolov8n-obb there is 9 end_node can be selected and postprocess it. let me make it simple in my case i take only the 3 conv end-node which is to get the angle as the image below shows: ONNX

The question is when i run both .ONNX and .HEF just to get the angle i should get the same output, right? Even after the optimizing the .HAR file there could be some change in the output but it will be slightly right not a dramatic different. I optimize the .HAR with resizing the dataset only this will be my code

images_path = “…/data/overlap”
images_list = [img_name for img_name in os.listdir(images_path) if os.path.splitext(img_name)[1] == “.jpg”]
calib_dataset = np.zeros((len(images_list), 640, 640, 3))

for idx, img_name in enumerate(sorted(images_list)):
img = Image.open(os.path.join(images_path, img_name))
img = img.resize((640, 640)) # Resize the image to 640x640
img = np.array(img)
calib_dataset[idx, :, :, :] = img

np.save(“calib_set.npy”, calib_dataset)

*For your information i already postprocess the output from the 3 node which is a simple step reshape-> concat → sigmoid → subtract(0.25) → multiply(PI) → angel

saurabh · October 8, 2024, 12:27pm

I also want to test the obb yolov8n obb model. But there is no default cfg yaml,alls and post processing files for obb in model-zoo. It would be great if you can share resouces. Or if you can explain. I will try.

saurabh · October 8, 2024, 12:28pm

@SAN

SAN · October 9, 2024, 1:17am

Hi @saurabh as i know there is a model called yolov8n_obb from ultralytics which what they do for the detection is similar with the yolov8n structure the only difference is where at last detection layer will be process with angel detection. This is the basic structure of the model. Right what we have to do is based on my understanding:

Choose the end node based on how you will postprocess it. In my case what i did was i folloe HAILO mz where hailo choose 6 conv node as their end node then postprocess it For us i will choose additional 3 conv node which means i will be having 9 end node. This is the 3 conv end node for angle
For the postprocessing part we need to apply RNMS in which you can add on the runner But for testing what i did was just run the hailo converted model .HEF on hailo, Get the raw output then post process it on my cpu. By this you can make sure that the model raw output is correct with the default model(.pt , .onnx)

runner.load_model_script(alls)

For the optimization what i did was simple resizing to 640 x 640 and optimize with data calibration dataset this is the code:

images_path = “…/data/overlap”
images_list = [img_name for img_name in os.listdir(images_path) if os.path.splitext(img_name)[1] == “.jpg”]

calib_dataset = np.zeros((len(images_list), 640, 640, 3))
for idx, img_name in enumerate(sorted(images_list)):
img = Image.open(os.path.join(images_path, img_name))
img = img.resize((640, 640)) # Resize the image to 640x640
img = np.array(img)
calib_dataset[idx, :, :, :] = img

np.save(“calib_set.npy”, calib_dataset)
model_name = “best”
hailo_model_har_name = f"{model_name}_hailo_model9.har"
assert os.path.isfile(hailo_model_har_name), “Please provide valid path for HAR file”
runner = ClientRunner(har=hailo_model_har_name)

runner.optimize(calib_dataset)

Save the result state to a Quantized HAR file

quantized_model_har_path = f"{model_name}_quantized_model.har"
runner.save_har(quantized_model_har_path)

Problem:

i manage to optimize it then converted to .har → .hef but the output im get from the convolution is not same with the ultralytics model. Even if my quantization is poor also the output should be only have some slight changes only but in my case its not even similar

SAN · October 9, 2024, 1:18am

If you manage to work on it, Please do share some idea on it @saurabh Thanks in advance

saurabh · October 9, 2024, 2:58am

@SAN
Thank you so much for the detailedexplanation, this is going to help a lot. I will try.

saurabh · October 13, 2024, 3:22pm

@SAN
I am able to create the hef for obb. But I am unable to test the result because of post processing.
Have you written post processing part? I am unable to build the post processing part for the outputs. I am unsure how to start, and where to find relevant information.
It would be great if you can share.
Thank you.

SAN · October 14, 2024, 1:34am

okkay yep sure maybe i know what is youre end node and how did you optimized it can you share it because for the post processing im still testing by using some code from ultralytics which you can get from the head module. But Without the end node there could be lots of way to post processing the raw output

SAN · October 14, 2024, 2:09am

Hi @saurabh i have post a new topic on a problem im facing with my pre processing code maybe it might helpfull for you .har vs .hef

saurabh · October 14, 2024, 2:25am

@SAN
Yes probably same 9 nodes. (3for angle). I used the pose alls script(ends nodes are same so). And I am using DOTA dataset to optimize quantization.

I also looked ultralytics head file, to understand what exactly written here.
Probably this what we need

github.com

ultralytics/ultralytics/blob/main/ultralytics/nn/modules/head.py#L200-L216


      
          def forward(self, x):
              """Concatenates and returns predicted bounding boxes and class probabilities."""
              bs = x[0].shape[0]  # batch size
              angle = torch.cat([self.cv4[i](x[i]).view(bs, self.ne, -1) for i in range(self.nl)], 2)  # OBB theta logits
              # NOTE: set `angle` as an attribute so that `decode_bboxes` could use it.
              angle = (angle.sigmoid() - 0.25) * math.pi  # [-pi/4, 3pi/4]
              # angle = angle.sigmoid() * math.pi / 2  # [0, pi/2]
              if not self.training:
                  self.angle = angle
              x = Detect.forward(self, x)
              if self.training:
                  return x, angle
              return torch.cat([x, angle], 1) if self.export else (torch.cat([x[0], angle], 1), (x[1], angle))
          
          def decode_bboxes(self, bboxes, anchors):
              """Decode rotated bounding boxes."""
              return dist2rbox(bboxes, self.angle, anchors, dim=1)

I am new to ml. And still alot of things i need to study. So, for me it seems little difficult to proceed. Although i will try. I
So, in case you can share it would be great and very helpful.

SAN · October 14, 2024, 2:35am

ohh that nice okay lemme give my idea on how i test it. for the preprocessing yes youre true ultralytics got some code which is in depth but i do have the pre processeing code in the google colab i gave you. The onnx code is on the google colab last section.

let me give some idea. what you see the angle = torch.cat … code that is this part

as i saw the angle code was quite easy to post process since it is sigmoid → sub(-0.25) → * pi then you will get the angle by this you can confirm the output angle data with the .pt/.onnx model

SAN · October 14, 2024, 2:38am

but the problem im facing is the .har file im able to get correct output as i test with the 3 conv node but when i convert to .hef im getting 2 issues where:

different input buffer but the same is correct which doesn’t make sense
even i make the input buffer size same aswell as the .har file still the out are note same at all.

hope you could get solution @saurabh

saurabh · October 14, 2024, 3:32am

@SAN
Thank you so much. I will try.

Just for confirmation –
I can use this code from pose-process to extract boxes…

github.com

hailo-ai/Hailo-Application-Code-Examples/blob/4073f8c252f07c95b6ee03c5bbc810835a9fdd04/runtime/python/pose_estimation/pose_estimation_utils.py#L351-L378


      
          for box_distribute, kpts, stride, _ in zip(raw_boxes, raw_kpts, strides, np.arange(3)):
              shape = [int(x / stride) for x in image_dims]
              grid_x = np.arange(shape[1]) + 0.5
              grid_y = np.arange(shape[0]) + 0.5
              grid_x, grid_y = np.meshgrid(grid_x, grid_y)
              ct_row = grid_y.flatten() * stride
              ct_col = grid_x.flatten() * stride
              center = np.stack((ct_col, ct_row, ct_col, ct_row), axis=1)
          
              reg_range = np.arange(reg_max + 1)
              box_distribute = np.reshape(box_distribute,
                                          (-1,
                                          box_distribute.shape[1] * box_distribute.shape[2],
                                          4,
                                          reg_max + 1))
              box_distance = self._softmax(box_distribute) * np.reshape(reg_range, (1, 1, 1, -1))
              box_distance = np.sum(box_distance, axis=-1) * stride
          
              box_distance = np.concatenate([box_distance[:, :, :2] * (-1), box_distance[:, :, 2:]],
                                          axis=-1)

This file has been truncated. show original

And I will have to calculate angle separately. And process the boxes and angle using

github.com

ultralytics/ultralytics/blob/b89d6f407044276b1f54753ef98c719e89928631/ultralytics/utils/tal.py#L336-L355


      
          def dist2rbox(pred_dist, pred_angle, anchor_points, dim=-1):
              """
              Decode predicted rotated bounding box coordinates from anchor points and distribution.
          
              Args:
                  pred_dist (torch.Tensor): Predicted rotated distance, shape (bs, h*w, 4).
                  pred_angle (torch.Tensor): Predicted angle, shape (bs, h*w, 1).
                  anchor_points (torch.Tensor): Anchor points, shape (h*w, 2).
                  dim (int, optional): Dimension along which to split. Defaults to -1.
          
              Returns:
                  (torch.Tensor): Predicted rotated bounding boxes, shape (bs, h*w, 4).
              """
              lt, rb = pred_dist.split(2, dim=dim)
              cos, sin = torch.cos(pred_angle), torch.sin(pred_angle)
              # (bs, h*w, 1)
              xf, yf = ((rb - lt) / 2).split(1, dim=dim)
              x, y = xf * cos - yf * sin, xf * sin + yf * cos
              xy = torch.cat([x, y], dim=dim) + anchor_points
              return torch.cat([xy, lt + rb], dim=dim)

Not sure if it is correct. May be not because inputs are different. But I think the approach should be very similar?

saurabh · October 14, 2024, 3:42am

@SAN
I have one more question…
How can we test using har file? I am not sure about it?
I have compiler setup on ubuntu and hailo connected to pi.

SAN · October 14, 2024, 3:45am

as i see as i know some of the preprocessing/postprocessing is on the hailo sdk client you can see this when you start the Hailo AI Software Suite Container environment the code are in there. Hailo sdk. for example let me show you the part they postprocessing for yolov8n

this is the script i get from the hailo_sdk_client you ca get form the docker container inside and get some idea on it

Topic		Replies	Views
Obb model quantization poor benchmark General network	2	49	November 5, 2024
General questions about hailo8l and object detection models General python , hailo8	2	50	November 4, 2024
Model Zoo Object Detection Yolov10 AccessDenied General	1	58	September 2, 2024
Compilation of 1920 imgsz model General	1	82	August 9, 2024
[Python API][RPI5] Coherent inference boxes but not their positions General raspberry-pi , python , hailo8 , hailo-api	6	74	August 26, 2024

Is there any oriented bounding box model object detection

For the post-processing (HailoRT-PP) and quantization steps:

Next Steps:

Save the result state to a Quantized HAR file

Related topics