Trouble running custom yolo.hef models with imgz = 1088

Hi! I am having trouble running models that have been trained on image size 1088, compiled with hailomz on hailo devices. I use this command for compiling my models:

hailomz compile yolov8n --ckpt yolov8n_1088.onnx --hw-arch hailo8l --calib-path dataset_only_20_1088 --classes 8 --resize 1088

I have tried without the resize parameter as well. Also, I have tried with --performance flag. Please note that the calibration dataset that I provide is also 1088x1088 images. I get no errors while compiling.

When I run inference on my hailo device using the .hef model, the output tensors have bounding boxes outside of the boundaries. Example:

[0.27948668599128723, 1.413228988647461, 0.3169996738433838, 1.4515174627304077]
[0.2144680768251419, 1.666332721710205, 0.2797059118747711, 1.6999872922897339]
[0.41739657521247864, 1.6333305835723877, 0.4508034586906433, 1.664322853088379]
[0.6060263514518738, 1.0992581844329834, 0.6771869659423828, 1.180235743522644]
[0.6399933695793152, 1.6056666374206543, 0.7100033164024353, 1.6840683221817017]
[0.7627313137054443, 1.8049031496047974e-05, 1.0805031061172485, 0.10052971541881561]
[0.5417605638504028, -0.0013944879174232483, 0.7329568862915039, 0.2380548119544983]
[1.1810753345489502, 0.14888127148151398, 1.2470234632492065, 0.17751646041870117]
[1.1512645483016968, 0.17490966618061066, 1.218542218208313, 0.19973406195640564]
[1.151106357574463, 0.18052802979946136, 1.2170072793960571, 0.20563937723636627]

As you can see, some points are above 1, and some below 0. I have also tried printing the raw tensor outputs from the model and they show the same thing. What could be the problem? Do I compile the model wrong? Models trained on 640 images work perfectly fine

Hey @michael.nilsson ,

I understand that the issue with bounding boxes exceeding the [0, 1] range is happening specifically with the 1088x1088 configuration while working fine at 640x640. Let me help you resolve this.

Understanding the Issue

The problem typically occurs when there’s a mismatch between:

  • Training preprocessing and inference preprocessing
  • How the model was compiled for the target resolution
  • How padding and scaling are handled during inference

Solution Steps

1. Check Your Training Preprocessing

If you’re using YOLOv8, verify your preprocessing matches this pattern:

from ultralytics.yolo.utils import ops

def preprocess(image, target_size=1088):
    # Resize and pad while maintaining aspect ratio
    image, ratio, (dw, dh) = ops.letterbox(image, new_shape=(target_size, target_size))
    # Normalize to [0, 1]
    image = image / 255.0
    return image, ratio, dw, dh

2. Adjust Your Compilation Command

Remove the --resize parameter if your model was specifically trained for 1088x1088:

hailomz compile yolov8n --ckpt yolov8n_1088.onnx \
    --hw-arch hailo8l \
    --calib-path dataset_only_20_1088 \
    --classes 8

3. Handle Output Scaling

If you’re using padding (letterboxing), make sure to adjust the bounding boxes:

def scale_boxes(img_shape, boxes, ratio, dw, dh):
    # Adjust for padding
    boxes[:, [0, 2]] -= dw  # x-coordinates
    boxes[:, [1, 3]] -= dh  # y-coordinates
    # Scale to original size
    boxes[:, [0, 2]] /= ratio
    boxes[:, [1, 3]] /= ratio
    return boxes

If you’re still experiencing issues after trying these steps, please share:

  1. Your training preprocessing code
  2. The exact compilation command you’re using
  3. A sample of the problematic output values

Best regards,
Omria