Converted Model Accuracy Issue

General
1

so I followed this guide to convert my custom retrained onnx model to hef, the only difference is I tried doing it on Docker, using Hailo AI Software Suite version 2025-01, the docker is running on windows 11.

The conversion runs fine, and I found no errors while running the scripts, however, when I test the converted hef model on hailo8l (raspi 5 + raspi ai kit), the performance is really bad, it can’t detect anything, apart from the dataset video (I tested it, in case it was a problem with the input), I thought it was a problem with my calibration data being too little, so I increased it to about 600 images, this doesn’t solve the problem, the converted model still can’t detect anything, then i noticed the “[info] Using dataset with 64 entries for calibration” message, so I reduced the calibration data, but diversify it, this leads to the model, not being able to detect anything at all, even when fed with the dataset video

I also checked my inference script, im currently using the one from the older version of Hailo-Application-Code-Examples, the yolox_stream_inference.py script, i changed the post processing a bit, and its running fine with yolov8 models from the model zoo, so im not sure what causes this problem

I also run hailo profiler command, and opened the .html file, but im not sure what to do with the information, does the optimization changes the weights of the model? because the onnx model runs fine

2

you should try analyse each step in the process incrementally. For example, you parse your onnx file to .har, what are the metrics like on the validation set with this model? If that works ok, then try the same with the optimized model (assuming you’ve done so). And so on.

If your parsed .har model isn’t able to detect anything, your compiled .hef won’t either. So you need to work out which step the inaccuracies occur.

My recommendation for each step, if you get bad metrics, is to take some sample images and run inference on that single image to check the outputs. You may find some insight there, e.g. maybe you need to adjust your rescaling factor or something.

1 Like
3

thx for the advice, i didn’t know har file can be used for inference before

after some digging, i found the emulator part on the DFC documentation page 76, but im not sure how to use it, i didn’t find what the input data supposed to be, so i just put in an image as array

Screenshot 2025-03-31 173018
Screenshot 2025-03-31 1730181288×344 19.2 KB

then i got this error:

Inference: 0entries [00:00, ?entries/s]Traceback (most recent call last):
File “/local/workspace/model_conversion_v2/har_infer.py”, line 14, in
output = runner.infer(ctx, np.asarray(image))
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_common/states/states.py”, line 16, in wrapped_func
return func(self, *args, **kwargs)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 373, in infer
return self.infer_emulator(infer_info)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/runner/client_runner.py”, line 401, in infer_emulator
return self.sdk_backend.acceleras_inference(infer_info)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_sdk_client/sdk_backend/sdk_backend.py”, line 1265, in acceleras_inference
return self.inference_flow.run(infer_info.data, batch_size, infer_info.data_count)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/flows/inference_flow.py”, line 65, in run
infer_output = self.predict(dataset, callbacks=[callback], verbose=0)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/tmp/autograph_generated_fileu8lkqd84.py", line 15, in tf__predict_function
retval = ag.converted_call(ag.ld(step_function), (ag_.ld(self), ag__.ld(iterator)), None, fscope)
File "/tmp/autograph_generated_fileu3alfnub.py", line 12, in tf__call
retval = ag_.converted_call(ag__.ld(self).model, (ag_.ld(inputs),), dict(**ag__.ld(kwargs)), fscope)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/utils/distributed_utils.py”, line 122, in wrapper
res = func(self, *args, **kwargs)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py”, line 1167, in build
layer.build(layer.input_shapes)
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 1552, in build
self._verify_and_set_hn_io_shapes()
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_io.py”, line 224, in _verify_and_set_hn_io_shapes
super()._verify_and_set_hn_io_shapes()
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 1665, in _verify_and_set_hn_io_shapes
if not self._verify_hn_to_keras_shapes(self._get_hn_input_shapes(), input_shapes):
File “/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py”, line 1692, in _verify_hn_to_keras_shapes
if not (np.array(keras_shape) == np.array(hn_shape)).all():
AttributeError: in user code:

File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/engine/training.py", line 2169, in predict_function  *
    return step_function(self, iterator)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/engine/training.py", line 2155, in step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/engine/training.py", line 2143, in run_step  **
    outputs = model.predict_step(data)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/engine/training.py", line 2111, in predict_step
    return self(x, training=False)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
File "/tmp/__autograph_generated_fileu3alfnub.py", line 12, in tf__call
    retval_ = ag__.converted_call(ag__.ld(self)._model, (ag__.ld(inputs),), dict(**ag__.ld(kwargs)), fscope)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/utils/distributed_utils.py", line 122, in wrapper
    res = func(self, *args, **kwargs)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py", line 1167, in build
    layer.build(layer.input_shapes)
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1552, in build
    self._verify_and_set_hn_io_shapes()
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_io.py", line 224, in _verify_and_set_hn_io_shapes
    super()._verify_and_set_hn_io_shapes()
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1665, in _verify_and_set_hn_io_shapes
    if not self._verify_hn_to_keras_shapes(self._get_hn_input_shapes(), input_shapes):
File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1692, in _verify_hn_to_keras_shapes
    if not (np.array(keras_shape) == np.array(hn_shape)).all():

AttributeError: Exception encountered when calling layer 'simulation_inference_model' (type SimulationInferenceModel).

in user code:

    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/flows/inference_flow.py", line 140, in call  *
        return self._model(inputs, **kwargs)
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler  **
        raise e.with_traceback(filtered_tb) from None
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/utils/distributed_utils.py", line 122, in wrapper
        res = func(self, *args, **kwargs)
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/model/hailo_model/hailo_model.py", line 1167, in build
        layer.build(layer.input_shapes)
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1552, in build
        self._verify_and_set_hn_io_shapes()
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/hailo_io.py", line 224, in _verify_and_set_hn_io_shapes
        super()._verify_and_set_hn_io_shapes()
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1665, in _verify_and_set_hn_io_shapes
        if not self._verify_hn_to_keras_shapes(self._get_hn_input_shapes(), input_shapes):
    File "/local/workspace/hailo_virtualenv/lib/python3.10/site-packages/hailo_model_optimization/acceleras/hailo_layers/base_hailo_layer.py", line 1692, in _verify_hn_to_keras_shapes
        if not (np.array(keras_shape) == np.array(hn_shape)).all():

    AttributeError: 'bool' object has no attribute 'all'


Call arguments received by layer 'simulation_inference_model' (type SimulationInferenceModel):
  • inputs=tf.Tensor(shape=(None, 1024, 3), dtype=float32)
  • kwargs={'training': 'False'}

Inference: 0entries [00:00, ?entries/s]

4

I found the problem, the input color needs to be converted, im using
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

so I when i was looking for an inference script, I came across this topic , then I modified the script a bit, I tested the HAR file and it worked flawlessly, then i noticed the color conversion, took me 5 weeks to figure this out…

here’s my host machine inference script if anyone’s interested

import os
import cv2
import numpy as np
from hailo_sdk_client import ClientRunner,InferenceContext

img_w = 1024
img_h = 1024

count = 0
validation_image_path = ‘validation_images/logo_detector_v1’
inference_result_path = ‘host_infer_result/logo-detector_yolov8s’
for file in os.listdir(validation_image_path):
if file.lower().endswith(‘.jpg’):
count+=1
print(f’processing image {count}')
img = cv2.imread(os.path.join(validation_image_path, file), cv2.IMREAD_COLOR)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# img = img / 255.0
img = img.astype(np.float32)
img = np.array([img])

    model_name = "logo-detector_yolov8s"
    har_path = f"har_files/{model_name}_quantized_model.har"
    runner = ClientRunner(har=har_path)

    with runner.infer_context(InferenceContext.SDK_QUANTIZED) as ctx:
        output = runner.infer(ctx, np.array(img))
    xmin = int(output[0][0][1][0] * img_w)
    ymin = int(output[0][0][0][0] * img_h)
    xmax = int(output[0][0][3][0] * img_w)
    ymax = int(output[0][0][2][0] * img_h)

    img = img[0]
    img = (img * 255).astype(np.uint8)
    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    img = cv2.bitwise_not(img)
    img = cv2.rectangle(img, (xmin, ymin), (xmax, ymax), (255,0,0), thickness=2)

    print(img.shape, img.dtype)

    if not cv2.imwrite(f"{inference_result_path}/result{count}.jpg",img):
        raise Exception("image not written")