hailomz eval errors using VisDrone dataset

natsayin_nahin · March 12, 2025, 10:31am

I have a yolo11n model finetuned on visdrone.

I downloaded the validation dataset manually then ran:

python create_visdrone_tfrecord.py -d=/local/shared_with_docker/VisDrone2019-DET-val val

which gave me the following output:

2025-03-10 22:30:22.796868: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-10 22:30:22.821680: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-03-10 22:30:23.182437: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
  0%|                                                                                                                                           | 0/548 [00:00<?, ?it/s]/local/workspace/hailo_model_zoo/hailo_model_zoo/datasets/create_visdrone_tfrecord.py:128: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
  "mask": _bytes_feature(np.array(0).tostring()),
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548/548 [00:00<00:00, 2008.95it/s]
Finished writing, skipped 1378 annotations.
38791 Annotations Found.
0 images are missing annotations.

I try to run using the detail models for yolo11n just to test the dataset, but run into this error:

!hailomz eval yolov11n --data-path=/local/shared_with_docker/.hailomz/models_files/visdrone/2020-05-25/visdrone_val.tfrecord

the full output is here:

<Hailo Model Zoo INFO> Start run for network yolov11n ...
<Hailo Model Zoo INFO> Initializing the runner...
<Hailo Model Zoo INFO> Chosen target is full_precision
[info] Translation started on ONNX model yolov11n
[info] Restored ONNX model yolov11n (completion time: 00:00:00.04)
[info] Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.17)
[info] NMS structure of yolov8 (or equivalent architecture) was detected.
[info] In order to use HailoRT post-processing capabilities, these end node names should be used: /model.23/cv3.0/cv3.0.2/Conv /model.23/cv2.0/cv2.0.2/Conv /model.23/cv3.1/cv3.1.2/Conv /model.23/cv2.1/cv2.1.2/Conv /model.23/cv2.2/cv2.2.2/Conv /model.23/cv3.2/cv3.2.2/Conv.
[info] Start nodes mapped from original model: 'images': 'yolov11n/input_layer1'.
[info] End nodes mapped from original model: '/model.23/cv2.0/cv2.0.2/Conv', '/model.23/cv3.0/cv3.0.2/Conv', '/model.23/cv2.1/cv2.1.2/Conv', '/model.23/cv3.1/cv3.1.2/Conv', '/model.23/cv2.2/cv2.2.2/Conv', '/model.23/cv3.2/cv3.2.2/Conv'.
[info] Translation completed on ONNX model yolov11n (completion time: 00:00:00.63)
[info] Appending model script commands to yolov11n from string
[info] Added nms postprocess command to model script.
[info] Saved HAR to: /local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_tutorials/notebooks/yolov11n.har
<Hailo Model Zoo INFO> Preparing calibration data...
[info] Loading model script commands to yolov11n from /local/workspace/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov11n.alls
<Hailo Model Zoo INFO> Initializing the dataset ...
<Hailo Model Zoo INFO> Running inference...
[info] Setting NMS score threshold to 0.001
Processed: 0images [00:01, ?images/s]
Traceback (most recent call last):
  File "/local/workspace/hailo_virtualenv/bin/hailomz", line 33, in <module>
    sys.exit(load_entry_point('hailo-model-zoo', 'console_scripts', 'hailomz')())
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main.py", line 122, in main
    run(args)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main.py", line 111, in run
    return handlers[args.command](args)
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/main_driver.py", line 403, in evaluate
    return infer_model_tf2(
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/core/main_utils.py", line 522, in infer_model_tf2
    return infer_callback(
  File "/local/workspace/hailo_model_zoo/hailo_model_zoo/core/infer/model_infer.py", line 58, in model_infer
    for preprocessed_data, img_info in batched_dataset:
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 797, in __next__
    return self._next_internal()
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 780, in _next_internal
    ret = gen_dataset_ops.iterator_get_next(
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 3016, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 7262, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__IteratorGetNext_output_types_15_device_/job:localhost/replica:0/task:0/device:CPU:0}} Paddings must be non-negative: 0 -27
	 [[{{node PadV2_1}}]] [Op:IteratorGetNext]
/local/shared_with_docker/.hailomz/models_files/visdrone/2020-05-25/visdrone_val.tfrecord'

omria · March 16, 2025, 4:29pm

Hey @natsayin_nahin ,

The error “Paddings must be non-negative: 0 -27” occurs during dataset preprocessing. This indicates an issue with padding calculations in your TensorFlow operations.

This is likely caused by:

Mismatched image dimensions in your dataset
Calculation errors in padding values
Issues in your TFRecord file structure

We recommend:

Verify all images have dimensions compatible with the model’s input requirements
Review your create_visdrone_tfrecord.py script for correct TFRecord generation
Ensure all padding operations use non-negative values
Add debugging code to print image shapes before padding operations

As a possible solution, you could try using the Hailo Model Zoo’s provided scripts for creating properly formatted TFRecord files:

python hailo_model_zoo/datasets/create_visdrone_tfrecord.py train -d /path/to/VisDrone2019-DET-train
python hailo_model_zoo/datasets/create_visdrone_tfrecord.py val -d /path/to/VisDrone2019-DET-val

These scripts are designed to convert the dataset into a format suitable for evaluation, optimization, and compilation with our tools.

If you continue experiencing issues, please share your preprocessing code for more specific guidance.

natsayin_nahin · March 18, 2025, 5:53am

Hi @omria, not sure if it wasn’t clear from the information provided - but I am using the hailo model zoon script, that is generating the very tfrecord file that causes the eval to fail.

The only difference in my cli command previously mentioned and your suggested one, was that I ran mine from within the hailo_model_zoo/datasets directory

So unless there are some additional preprocessing steps that need to be done on the visdrone dataset, then the issue surely has to be in the script? Since I’m not doing anything custom here

Topic		Replies	Views
Possible bug in hailo eval <model_name. when using default datasets? General debug , hailo8	1	14	March 10, 2025
Facing error while running hailo applications code example git repo General hailort , hailo8	18	675	July 2, 2024
compilation error General hailo8	3	19	March 3, 2025
Trouble running custom yolo.hef models with imgz = 1088 General hailo8	1	28	January 27, 2025
hailomz compile: ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported. General	6	151	December 16, 2024

hailomz eval errors using VisDrone dataset

Related topics