4.19 Driver Error

Hello,

I wrote a gstreamer app in python which abruptly driver error working for 2 hours. Sometimes in 10 minutes sometimes 4 hours, the duration will change but definitely an throws the same error.

I am using hailo8 and driver 4.19, I am at ai suite-docker 3.29 on Ubuntu 22.04 on a kernel 5.19.0-1030-lowlatency

hailortcli fw-control identify
[HailoRT] [error] CHECK failed - Failed to open device file /dev/hailo0 with error 6
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)

And my app has this output:
[HailoRT] [warning] Channel 0:2 completed with errors: host_error 32 device_error 0
[HailoRT] [error] Infer request callback failed with status = HAILO_INTERNAL_FAILURE(8)
kafka_dict: {‘timestamp’: ‘2025-01-24 15:02:13’, ‘detections’: [{‘label’: ‘person’, ‘bbox’: [131.85245513916016, 236.5789794921875, 248.27640533447266, 396.0777282714844], ‘confidence’: 0.7328890562057495}, {‘label’: ‘person’, ‘bbox’: [69.07393455505371, 139.29180145263672, 129.45549964904785, 290.00221252441406], ‘confidence’: 0.6380507946014404}, {‘label’: ‘person’, ‘bbox’: [203.1374168395996, 271.75275802612305, 260.92369079589844, 362.2074508666992], ‘confidence’: 0.5198192596435547}], ‘stream_id’: ‘cameraid1’, ‘model_name’: ‘yolov5’}
[HailoRT] [error] CHECK failed - Failed launch transfer errno: 14
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK failed - Failed launch transfer errno: 14
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK failed - Failed launch transfer errno: 14
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [warning] Failed to acquire buffer because the buffer pool is empty. This could be caused by uneven reading and writing speeds, with a short user-defined timeout. (timeout=10000ms)
[HailoRT] [error] Non-recoverable Async Infer Pipeline error. status error code: HAILO_TIMEOUT(4)
[HailoRT] [error] Shutting down the pipeline with status HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK failed - Failed in fw_control, errno:19
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to send fw control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to disable state machine
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to set state machine
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Scheduler failed deactivate core op on 0000:03:00.0
[HailoRT] [error] Scheduler failed deactivate core op on 0000:03:00.0
[HailoRT] [error] CHECK failed - Got timeout in wait_for_async_ready (10000ms) - the edge ‘yolov5m_wo_spp_60p/yolov5_nms_postprocess’ could not receive 1 transfer-requests
CHECK_SUCCESS failed with status=4
CHECK_SUCCESS failed with status=4
Error: gst-stream-error-quark: Internal data stream error. (1), …/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstRTSPSrc:source_2/GstUDPSrc:udpsrc0:
streaming stopped, reason error (-5)
[HailoRT] [error] CHECK failed - Failed shutdown, ongoing infer requests - 1
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [critical] Executing pipeline terminate failed with status HAILO_TIMEOUT(4)

Hi @jackson,

I notice you’re experiencing multiple communication errors between your host hardware and the Hailo-8. Let me break down what we’re seeing:

  1. Driver/Firmware Communication Issues:

    • HAILO_DRIVER_FAIL(36)
    • HAILO_FW_CONTROL_FAILURE(18)
  2. Resource Management Problems:

    • Failed to acquire buffer because the buffer pool is empty
    • Non-recoverable Async Infer Pipeline error
  3. Pipeline Overload Issues:

    • CHECK failed - Failed launch transfer errno: 14
    • HAILO_TIMEOUT(4)

Here’s what I recommend checking:

  1. Update your kernel to a version between 5.5 and 6.5 for better stability
  2. Verify all physical connections to your Hailo device
  3. Important: Check if the thermal pad is properly installed on/under your Hailo device

These hardware and system-level checks should help resolve the communication issues you’re experiencing. If the issue continue , please provide the following :
1- full log
2-run reboot , then run the following : hailortcli scan
hailortcli fw-control identify

Let me know if you need any clarification on these steps!

Best regards,
Omria