Timeout and Buffer Pool Warnings When Running rpiExamples on Raspberry Pi 5 with Hailo-8 (26 TOPS)

General
, , ,
1

Hi Hailo Community,

I’m encountering an issue while running the rpiExamples on a Raspberry Pi 5 with the Hailo-8 PCIe module (26 TOPS). I’m following the instructions exactly as described in the documentation, and I’m using the official Raspberry Pi 5 power supply (5V/5A).

After a few hundred frames of successful detection, I start receiving the following warnings and errors:
the comand is:

python basic_pipelines/detection.py --input usb

Frame count: 295
Detection: ID: 1 Label: person Confidence: 0.95

Frame count: 300
Detection: ID: 1 Label: person Confidence: 0.96

[HailoRT] [warning] Failed to acquire buffer because the buffer pool is empty. This could be caused by uneven reading and writing speeds, with a short user-defined timeout. (timeout=10000ms)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Failed to acquire buffer with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK failed - PreInferEl1yolov8m/input_layer1 (H2D) failed with status=HAILO_TIMEOUT(4) (timeout=10000ms)
[HailoRT] [error] CHECK failed - Got timeout in `wait_for_async_ready` (10000ms) - the edge 'yolov8m/yolov8_nms_postprocess' could not receive 1 transfer-requests

dmesg | grep -i hailo

[    3.130849] hailo: Init module. driver version 4.20.0
[    3.130985] hailo 0001:01:00.0: Probing on: 1e60:2864...
[    3.130992] hailo 0001:01:00.0: Probing: Allocate memory for device extension, 13184
[    3.131011] hailo 0001:01:00.0: enabling device (0000 -> 0002)
[    3.131018] hailo 0001:01:00.0: Probing: Device enabled
[    3.131045] hailo 0001:01:00.0: Probing: mapped bar 0 - 000000000ec16bcd 16384
[    3.131053] hailo 0001:01:00.0: Probing: mapped bar 2 - 000000003a646a55 4096
[    3.131056] hailo 0001:01:00.0: Probing: mapped bar 4 - 00000000fdf8f448 16384
[    3.131059] hailo 0001:01:00.0: Probing: Force setting max_desc_page_size to 4096 (recommended value is 16384)
[    3.131072] hailo 0001:01:00.0: Probing: Enabled 64 bit dma
[    3.131074] hailo 0001:01:00.0: Probing: Using userspace allocated vdma buffers
[    3.131080] hailo 0001:01:00.0: Disabling ASPM L0s
[    3.131085] hailo 0001:01:00.0: Successfully disabled ASPM L0s
[    3.131209] hailo 0001:01:00.0: Writing file hailo/hailo8_fw.bin
[    3.258164] hailo 0001:01:00.0: File hailo/hailo8_fw.bin written successfully
[    3.258171] hailo 0001:01:00.0: Writing file hailo/hailo8_board_cfg.bin
[    3.258195] Failed to write file hailo/hailo8_board_cfg.bin
[    3.258197] hailo 0001:01:00.0: File hailo/hailo8_board_cfg.bin written successfully
[    3.258198] hailo 0001:01:00.0: Writing file hailo/hailo8_fw_cfg.bin
[    3.258205] Failed to write file hailo/hailo8_fw_cfg.bin
[    3.258206] hailo 0001:01:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[    3.356565] hailo 0001:01:00.0: NNC Firmware loaded successfully
[    3.356570] hailo 0001:01:00.0: FW loaded, took 225 ms
[    3.368738] hailo 0001:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0
[  561.666395] hailo 0001:01:00.0: Failed writing fw control to pcie
[ 1071.016306] hailo 0001:01:00.0: hailo_nnc_driver_down, timeout waiting for shutdown response (timeout_ms=5)

It seems like the inference pipeline hits a bottleneck and the buffer pool is depleted, leading to timeouts.

My setup:

  • Raspberry Pi 5 (8GB)
  • Hailo-8 PCIe module (26 TOPS)
  • Official Raspberry Pi 5 power supply
  • Running basic_pipelines/detection.py via:
    python3 basic_pipelines/detection.py --input usb

Has anyone encountered this issue?
Is there a recommended way to:

  • Limit the frame rate to avoid overwhelming the pipeline?
  • Tune internal buffer sizes or timeouts?
  • Properly reset or recover the PCIe device after this happens?

Thanks in advance for your help!