HAILO_TIMEOUT Error

General
, ,
1

To run a custom Torch model on the Hailo8L hardware I performed the following steps.

  • Convert Torch model to ONNX
  • Convert ONNX model to HAR
  • Quantize HAR model
  • Compile HAR model to HEF

The HEF model has 1 input and 3 outputs as shown below.

(hailo) C:\Users\LattePanda\Desktop\hailo>hailo parse-hef model.hef
(hailo) Running command 'parse-hef' with 'hailortcli'
Architecture HEF was compiled for: HAILO8L
Network group name: model, Multi Context - Number of contexts: 4
    Network name: model/model
        VStream infos:
            Input  model/input_layer1 UINT8, NHWC(320x320x1)
            Output model/fc2 UINT8, NC(320)
            Output model/fc1 UINT8, NC(320)
            Output model/resize3 UINT8, F8CR(320x320x1)

When benchmarking the model I do however get the following error.

(hailo) C:\Users\LattePanda\Desktop\hailo>hailo benchmark model.hef
(hailo) Running command 'benchmark' with 'hailortcli'
Starting Measurements...
Measuring FPS in HW-only mode
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for output stream buffer model/fc2
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for output stream buffer model/resize3
[HailoRT] [error] CHECK_SUCCESS failed with status=0000:03:00.0 - Failed read from stream (device: HAILO_TIMEOUT(4))
[HailoRT] [error] CHECK_SUCCESS failed with status=0000:03:00.0 - Failed read from stream (device: HAILO_TIMEOUT(4))
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for input stream buffer model/input_layer1
[HailoRT] [error] CHECK_SUCCESS failed with status=0000:03:00.0 - Failed write to stream (device: HAILO_TIMEOUT(4))
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) -
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for output stream buffer model/fc1
[HailoRT] [error] CHECK_SUCCESS failed with status=0000:03:00.0 - Failed read from stream (device: HAILO_TIMEOUT(4))
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) -
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) -
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Error failed running inference
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Error while running inference
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Measuring FPS in HW-only mode failed

The compilation to HEF was successful, what could be the reason the HEF file now fails to run on the Hailo8L?

2

@m.wijkhuizen If you try a Hailo 8L model directly downloaded from the Hailo model zoo as sanity check, does it work ?

3

The problems seems to be that Torch models are channel fist and the converted HAR models are channels last. All native layers, such as Conv2D are correctly converted from channel first to channel last.
However, all custom linear algebra is not converted, causing runtime errors.
My current solution is to do only use native layers in the model and do all linear algebra post processing in a custom post-processing function that is not included in the model.

It would be great to maintain the channel first format in the HAR/HEF models to prevent these errors.

4

Hey @m.wijkhuizen

Here’s another approach you could try:

You can add Permute layers in your model to handle the channel format switching. These layers will convert between channel-first and channel-last formats as needed:

  • Add one at the start to convert to channel-last
  • Add another before your linear algebra ops to switch to channel-first
  • Convert back to channel-last afterwards if your other layers need it

This way, you can keep your custom operations while making sure everything stays aligned with the expected formats.

Let me know if you need help implementing this!