Hailort fw-control failed

I have encountered an error where the fw-control fails with my Hailo 8 M.2 M key. I did run the “hailortcli fw-control identify” command, and got this output:

Executing on device: 0000:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.18.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8
Serial Number: <N/A>
Part Number: <N/A>
Product Name: <N/A>

When I run lspci it does show up in the list of devices on my OrangePi 5 Plus (it did even before installing PCIe driver):

$ lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0000:01:00.0 Co-processor: Hailo Technologies Ltd. Hailo-8 AI Processor (rev 01)
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

It also shows up in /dev directory as /dev/hailo0, as well as in ‘hailortcli scan’

$ hailortcli scan
Hailo Devices:
[-] Device: 0000:01:00.0

I did try installing it with different methods which in some cases gives me an error where it fails to “open device file” with error 6 (when built from source) or 13 (on Arch linux with AUR hailort package). It also heats up quite a bit even when my SBC is turned off or/and when the Co-processor isn’t doing anything (because I can’t get it to run). What do I do in this case?

I did also attempt to benchmark a car detection .hef file found in Model_Zoo repo, but get this error relating to Hailort Firmware Control:

$ hailortcli benchmark /home/orangepi/Downloads/yolov5m_vehicles.hef
Starting Measurements…
Measuring FPS in hw_only mode
Network auto_group_0/yolov5m_vehicles: 100% | 1206 | FPS: 80.36 | ETA: 00:00:00
Measuring FPS and Power in streaming mode
[HailoRT] [error] Firmware control has failed. Major status: 0x400300b4, Minor status: 0x400300b4
[HailoRT] [error] Firmware major status: CONTROL_PROTOCOL_STATUS_UNSUPPORTED_DEVICE
[HailoRT] [error] Firmware minor status: CONTROL_PROTOCOL_STATUS_UNSUPPORTED_DEVICE
[HailoRT] [error] Opcode HAILO_CONTROL_OPCODE_SET_POWER_MEASUEMENT is not supported on the current board.
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_OPCODE(48)
Failed to set power measurement parameters, status HAILO_UNSUPPORTED_OPCODE(48)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_OPCODE(48) - Failed starting power measurement on device 0000:01:00.0
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_OPCODE(48) - Error while running inference
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_UNSUPPORTED_OPCODE(48) - FPS in streaming mode failed

Hey @isatay,

Welcome to the Hailo Community! Let’s address your questions in order:

  1. Installation recommendation:
  1. Driver installation errors:
  • Could you please provide more details about the specific errors you’re encountering during driver installation? This will help us offer more targeted assistance.
  1. Regarding heat management:
  • Did you place an insulation pad between the Hailo device and the M.2 board? This should typically be sufficient for heat management.
  • If you’re still experiencing heating issues, consider adding a heatsink for additional cooling.
  1. Fixing your current issue:
  • Run Hailortcli run <fileName.hef>
  • You can explore various options within this command to adjust settings like batch size and latency.

I did follow the instructions provided with .deb file, and it did install without an issue. And I did check and run the command “run <filename.hef>” , and it does seem to work, but then it fails when it comes to fw_control stuff. For example when I try to run “hailortcli fw-control identify” it doesn’t display Serial Number and some other information as you can see in the original message. I also tried getting tappas docker for aarch64, and running /apps/h8/gstreamer/rockchip/detection/detection.sh file, and for a split second it worked and even detected some objects, but it froze right away and then gave me an error relating to the fw-control:

Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for input stream buffer yolov5m_wo_spp_60p/input_layer1
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - HwWriteElement3_yolov5m_wo_spp_60p/input_layer1 (H2D) failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] Queue element PushQueueElement3_yolov5m_wo_spp_60p/input_layer1 run in thread function failed! status = HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
ERROR: from element /GstPipeline:pipeline0/GstHailoNet:hailonet0/GstHailoSend:hailosend0: Failed writing to input vstream yolov5m_wo_spp_60p/input_layer1, status = 4
Additional debug info:
/local/workspace/tappas/hailort/sources/hailort/libhailort/bindings/gstreamer/gst-hailo/gsthailosend.cpp(187): write_to_vstreams (): /GstPipeline:pipeline0/GstHailoNet:hailonet0/GstHailoSend:hailosend0
Execution ended after 0:00:10.051952914
Setting pipeline to NULL …
[HailoRT] [error] HAILO_FW_CONTROL failed with errno:19
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to send fw control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to reset context switch state machine
[HailoRT] [error] Error deactivating core-op when destroying scheduler
[HailoRT] [error] HAILO_FW_CONTROL failed with errno:19
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to send fw control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [warning] clear configured apps ended with status HAILO_FW_CONTROL_FAILURE(18)
Freeing pipeline …

could this issue with fw-control originate from the fact i’m using Easy EAI’s Hailo 8 M.2 board, or should your drivers work seamlessly with this board as well as with normal Hailo 8 boards?

Nevermind, I tried manually installing the Tappas from git, and it seems to work fine, despite the fw-control issues I see when I try to run “hailortcli fw-control identify”