HAILO_INTERNAL_FAILURE(8)

I have a custom application written in python that runs on a RPi5 with the 13 tops AI Hat (hailo-8l). The OS is Raspbian bookworm 64-bit with kernel

Linux raspberrypi 6.12.25+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64 GNU/Linux

and hailo versions

ii  hailo-all                             4.20.0                              all          Hailo support (metapackage)
ii  hailo-tappas-core                     3.31.0+1-1                          arm64        This package contains the core components of the Hailo Tappas platform.
ii  hailofw                               4.20.0-1                            all          Hailo firmware
ii  hailort                               4.20.0-1                            arm64        HailoRT
ii  python3-hailort                       4.20.0-1                            arm64        HailoRT Python API, which wraps the runtime library
ii  rpicam-apps-hailo-postprocess         1.7.0-1                             arm64        rpicam-apps-hailo

The application, at the moment, grabs a frame from a camera and then runs it through pose estimation before taking the results and doing some custom processing with them. It runs fine but periodically the app just exits without warning. The app’s logs don’t show any errors, just a sudden stop. When I look in the hailort.log file I see:

[2025-07-17 08:24:17.892] [2293] [HailoRT] [info] [pipeline.cpp:891] [print_deep_description] PushQEl8AsyncHwEl | inputs: AsyncHwEl[0] | outputs: PostInferEl8AsyncHwEl(running in thread_id: 210754)
[2025-07-17 08:24:17.892] [2293] [HailoRT] [info] [pipeline.cpp:891] [print_deep_description] PostInferEl8AsyncHwEl | inputs: PushQEl8AsyncHwEl[0] | outputs: LastAsyncEl0PostInferEl8AsyncHwEl
[2025-07-17 08:24:17.892] [2293] [HailoRT] [info] [pipeline.cpp:891] [print_deep_description] LastAsyncEl0PostInferEl8AsyncHwEl | inputs: PostInferEl8AsyncHwEl[0] | outputs: user
> [2025-07-17 08:24:17.927] [2293] [HailoRT] [error] [event.cpp:113] [eventfd_poll] poll failed with errno=4
> [2025-07-17 08:24:17.927] [2293] [HailoRT] [error] [event.cpp:55] [wait] CHECK_SUCCESS failed with status=HAILO_INTERNAL_FAILURE(8)

Can someone please advise what this failure is and how to avoid/handle it gracefully if at all possible? It is somewhat urgent to get this fixed.

Hi [Marc_Jasner],

Can you guide me :cry: how you install the libraries, hailo packages as in the picture.
i am downloading rpi5-exampes but it is giving error when ./install.sh .

:package: Installing missing pip packages

:package: Installing ‘hailort’ via helper script
→ HAILORT_VERSION = 4.20.0
→ TAPPAS_CORE_VERSION= 3.31.0
→ DOWNLOAD_DIR = hailo_temp_resources
→ install Tapas? = false
→ install HailoRT? = true
when i check dpkg -l | grep hailo
±-----------------------±-----------±-------±--------------------------------------------------------------+
| Package Name | Version | Arch | Description |
±-----------------------±-----------±-------±--------------------------------------------------------------+
| hailo-tappas-core | 3.31.0+1-1 | arm64 | Core components of the Hailo Tappas platform |
| hailort | 4.22.0 | arm64 | HailoRT runtime |
| hailort-pcie-driver | 4.22.0 | all | Hailo PCIe driver and firmware |

@Quan_Le_Minh,

I’m using a Raspberry Pi 5 so I installed the hailo software with “sudo apt install hailo-all”.

using only sudo apt install hailo-all”. ?
Have you tried downloading rpi5-example? Did you follow the github instructions or is there anything else you need to keep in mind, like about venv?

Hey @Marc_Jasner ,

Is this still an issue ?
Looking at those two log lines:

[event.cpp:113] [eventfd_poll] poll failed with errno=4
[event.cpp:55] [wait] CHECK_SUCCESS failed with status=HAILO_INTERNAL_FAILURE(8)

Here’s what’s happening:

The first line shows errno=4, which corresponds to EINTR - meaning the Linux poll() system call got interrupted by a signal. The second line shows HailoRT converting this into a HAILO_INTERNAL_FAILURE status (code 8), which is basically its catch-all for “something unexpected went wrong.”

The issue is that HailoRT’s event polling doesn’t automatically retry when this happens - it just treats any error as an internal failure.

How to fix it:

Add retry logic - Catch HAILO_INTERNAL_FAILURE and retry the operation.

Hope this helps!

Hey @Quan_Le_Minh ,

I can see you’ve got both HailoRT version 4.22 and 4.20 installed on your system, which is causing some confusion - the system doesn’t know which one to use.

I’d suggest sticking with version 4.20 for now and removing 4.22. Version 4.22 will be officially released with hailo-all soon, so it’s better to wait for that proper release rather than mixing versions.

This should clear up any conflicts you’re experiencing!

I will definitely look into adding a catch for HAILO_INTERNAL_FAILURE. I wasn’t aware you could catch that, but that’s good to know.

In the meanwhile I think I was able to resolve the problem. When I first installed the Hailo board a few months back I remember the instructions listing the PCIe Gen 3 setting as optional, but I see it’s now not marked optional. I tried enabling it and since then my unit has been fine, so I think upgrading to the newest kernel AND enabling PCIe Gen 3 was the fix for me.

Thanks!

1 Like

Is HAILO_INTERNAL_FAILURE an exception that can be caught in python, or only c++? My app is python.

One of my devices still occasionally has this issue, but im in the process of starting clean with that device to see if it clears up. Either way I’d like to be able to catch and handle that error in python if it does occur.

It looks like I can catch it with HailoRTException. Is that correct?

Are we sure the HAILO_INTERNAL_FAILURE is being raised as a python exception? I have a try except around the code that looks like this:

try:
hailo_inference.run()
preprocess.join()
# To signal processing process to exit
output_queue.put(None)
postprocess.join()

    check_process_errors(preprocess, postprocess)

except Exception as e:
    logger.error(f"Inference error: {e}")
    # Ensure cleanup if there's an error
    input_queue.close()
    output_queue.close()
    preprocess.terminate()
    postprocess.terminate()

    os._exit(1)  # Force exit on error

Shouldn’t the general Exception clause catch the HailoRTException? That seems to imply it’s not being raised up to Python so it can’t be caught by the app. Is that correct?

Tagging @omria to make sure it’s visible.

Hey @Marc_Jasner ,

HAILO_INTERNAL_FAILURE is a C++ status code from libhailort. The Python wrapper only converts it to a HailoRTException when the error crosses the Python-C++ boundary. In your case, the error occurs inside the C++ pipeline thread and only gets logged - it never propagates back to your Python code.

Your current code:

try:
    hailo_inference.run()  # starts async pipeline
    preprocess.join()
    postprocess.join()
except Exception as e:
    # This never executes because the error stays in C++
    ...

How to fix it

Option 1: Use synchronous API

try:
    configured_model.run([bindings], timeout_ms)
except HailoRTException as e:
    logger.error(f"Hailo error: {e}")
    # Handle the error properly

Option 2: Use async with callback

def on_complete(completion_info, _):
    if completion_info.exception:
        logger.error(f"Inference failed: {completion_info.exception}")
        # Handle the error

job = configured_model.run_async([bindings], callback=on_complete)

Option 3: Use async with explicit wait

job = configured_model.run_async([bindings])
try:
    job.wait(timeout_ms)
except HailoRTException as e:
    logger.error(f"Async job failed: {e}")

Key takeaway

  1. Yes, HailoRTException is the Python exception class for internal statuses (including HAILO_INTERNAL_FAILURE).
  2. No, your general except Exception won’t catch it unless you force the status back into Python—either by using the sync API or by explicitly checking completion_info.exception in an async callback.