Hi everyone!
I tried to run Hailo-8 on platform based on RISC-V architecture. I know it’s not officially supported, but I successfully implemented meta-hailo in my Yocto environment. All conditions from the subsection" Validating the Integration’s Success" are met.
Hailo-8 device is visible using lspci and hailortcli scan commands. I’m also able to measure power consumption using hailortcli measure-power command.
The problems start when I try to run .hef network or use benchmark on .hef file. Below are output logs from both commands. I used scdepthv3.hef file from one of the examples.
hailortcli run:
Running streaming inference (scdepthv3.hef):
Transform data: true
Type: auto
Quantized: true
Network scdepthv3/scdepthv3: 100% | 0 | FPS: 0.00 | ETA: 00:00:00
Inference result:
Network group: scdepthv3
Duration: 00:00:05
FPS: 0.00
Send Rate: 0.00 Mbit/s
Recv Rate: 0.00 Mbit/s
hailort.log file:
[2024-08-20 12:14:43.257] [517] [HailoRT] [info] [device.cpp:46] [Device] OS Version: Linux 5.15.68-linux4microchip+fpga-2022.09 #1 SMP Thu Sep 29 07:50:53 UTC 2022 riscv64
[2024-08-20 12:14:43.258] [517] [HailoRT] [info] [control.cpp:100] [control__parse_identify_results] firmware_version is: 4.18.1
[2024-08-20 12:14:43.259] [517] [HailoRT] [info] [vdevice.cpp:652] [create] VDevice Infos: 0000:04:00.0
[2024-08-20 12:14:43.296] [517] [HailoRT] [info] [internal_buffer_manager.cpp:204] [print_execution_results] Planned internal buffer memory: CMA memory 0, user memory 4194304. memory to edge layer usage factor is 1.0
[2024-08-20 12:14:43.296] [517] [HailoRT] [info] [internal_buffer_manager.cpp:212] [print_execution_results] Default Internal buffer planner executed successfully
[2024-08-20 12:14:43.534] [517] [HailoRT] [info] [device_internal.cpp:57] [configure] Configuring HEF took 274.741755 milliseconds
[2024-08-20 12:14:43.535] [517] [HailoRT] [info] [vdevice.cpp:751] [configure] Configuring HEF on VDevice took 275.715746 milliseconds
[2024-08-20 12:14:43.535] [517] [HailoRT] [info] [control.cpp:100] [control__parse_identify_results] firmware_version is: 4.18.1
[2024-08-20 12:14:43.536] [517] [HailoRT] [info] [core_op.cpp:156] [activate] Activating scdepthv3 took 0.665994 milliseconds. Note that the function is asynchronous and thus the network is not fully activated yet.
[2024-08-20 12:14:43.536] [517] [HailoRT] [info] [core_op.cpp:156] [activate] Activating scdepthv3 took 0.747994 milliseconds. Note that the function is asynchronous and thus the network is not fully activated yet.
[2024-08-20 12:14:43.538] [517] [HailoRT] [info] [edge_elements.cpp:52] [create] Created (HwWriteEl15scdepthv3/input_layer1 | hw_frame_size: 245760)
[2024-08-20 12:14:43.539] [517] [HailoRT] [info] [vstream.cpp:754] [InputVStreamImpl] Creating scdepthv3/input_layer1…
[2024-08-20 12:14:43.539] [517] [HailoRT] [info] [vstream_builder.cpp:102] [create_inputs] Input pipeline ‘scdepthv3/input_layer1’: (HwWriteEl15scdepthv3/input_layer1 | hw_frame_size: 245760) >> HW
[2024-08-20 12:14:43.539] [517] [HailoRT] [info] [edge_elements.cpp:287] [create] Created (HwReadEl12scdepthv3/conv31 | hw_frame_size: 163840)
[2024-08-20 12:14:43.540] [517] [HailoRT] [info] [queue_elements.cpp:756] [create] Created (UserBuffQEl12scdepthv3/conv31 | timeout: 10s)
[2024-08-20 12:14:43.540] [517] [HailoRT] [info] [vstream.cpp:1107] [OutputVStreamImpl] Creating scdepthv3/conv31…
[2024-08-20 12:14:43.540] [517] [HailoRT] [info] [vstream_builder.cpp:225] [create_outputs] Output pipeline ‘scdepthv3/conv31’: HW >> (HwReadEl12scdepthv3/conv31 | hw_frame_size: 163840) >> (UserBuffQEl12scdepthv3/conv31 | timeout: 10s)
[2024-08-20 12:14:48.545] [527] [HailoRT] [info] [vdevice_native_stream.cpp:142] [write_impl] Failed write to stream scdepthv3/input_layer1 (device: 0000:04:00.0) with status=HAILO_STREAM_ABORT(63)
[2024-08-20 12:14:48.545] [523] [HailoRT] [info] [vdevice_native_stream.cpp:331] [read_impl] Failed read from stream HAILO_STREAM_ABORT(63) (device: 0000:04:00.0)
[2024-08-20 12:14:48.545] [523] [HailoRT] [info] [edge_elements.cpp:396] [run_pull] Reading from stream was aborted!
[2024-08-20 12:14:48.545] [527] [HailoRT] [info] [edge_elements.cpp:87] [run_push] Failed to send on input stream InputStream(index=15, name=scdepthv3/input_layer1) because stream was aborted
[2024-08-20 12:14:48.545] [527] [HailoRT] [info] [vstream.cpp:778] [write] Sending to VStream was aborted!
[2024-08-20 12:14:48.545] [523] [HailoRT] [info] [queue_elements.cpp:822] [run_in_thread] run_pull of UserBuffQEl12scdepthv3/conv31 was aborted!
[2024-08-20 12:14:48.546] [526] [HailoRT] [info] [queue_elements.cpp:790] [run_pull] Shutdown event was signaled in dequeue of queue element UserBuffQEl12scdepthv3/conv31!
[2024-08-20 12:14:48.546] [526] [HailoRT] [info] [vstream.cpp:1131] [read] Receiving to VStream was shutdown!
[2024-08-20 12:14:48.551] [523] [HailoRT] [info] [queue_elements.cpp:54] [operator()] Thread in element UserBuffQEl12scdepthv3/conv31 is not running anymore, exiting…
[2024-08-20 12:14:48.551] [517] [HailoRT] [info] [queue_elements.cpp:43] [~BaseQueueElement] Queue element UserBuffQEl12scdepthv3/conv31 has 0 frames in his Queue on destruction
[2024-08-20 12:14:48.551] [517] [HailoRT] [info] [core_op.cpp:186] [deactivate] Deactivating took 0.026 ms
[2024-08-20 12:14:48.551] [517] [HailoRT] [info] [core_op.cpp:186] [deactivate] Deactivating took 0.106999 ms
hailortcli benchmark:
Starting Measurements…
Measuring FPS in hw_only mode
Network scdepthv3/scdepthv3: 60% | 0 | FPS: 0.00 | ETA: 00:00:06
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for output stream buffer scdepthv3/conv31
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Failed read from stream (device: 0000:04:00.0)
[HailoRT] [error] Got HAILO_TIMEOUT while waiting for input stream buffer scdepthv3/input_layer1
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Failed write to stream (device: 0000:04:00.0)
Network scdepthv3/scdepthv3: 93% | 0 | FPS: 0.00 | ETA: 00:00:01
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] Failed waiting for threads with status HAILO_TIMEOUT(4)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Error failed running inference
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Error while running inference
hailort.log file:
[2024-08-20 12:21:55.005] [596] [HailoRT] [info] [device.cpp:46] [Device] OS Version: Linux 5.15.68-linux4microchip+fpga-2022.09 #1 SMP Thu Sep 29 07:50:53 UTC 2022 riscv64
[2024-08-20 12:21:55.006] [596] [HailoRT] [info] [control.cpp:100] [control__parse_identify_results] firmware_version is: 4.18.1
[2024-08-20 12:21:55.007] [596] [HailoRT] [info] [vdevice.cpp:652] [create] VDevice Infos: 0000:04:00.0
[2024-08-20 12:21:55.044] [596] [HailoRT] [info] [internal_buffer_manager.cpp:204] [print_execution_results] Planned internal buffer memory: CMA memory 0, user memory 4194304. memory to edge layer usage factor >
[2024-08-20 12:21:55.044] [596] [HailoRT] [info] [internal_buffer_manager.cpp:212] [print_execution_results] Default Internal buffer planner executed successfully
[2024-08-20 12:21:55.299] [596] [HailoRT] [info] [device_internal.cpp:57] [configure] Configuring HEF took 290.891108 milliseconds
[2024-08-20 12:21:55.299] [596] [HailoRT] [info] [vdevice.cpp:751] [configure] Configuring HEF on VDevice took 291.898101 milliseconds
[2024-08-20 12:21:55.300] [596] [HailoRT] [info] [control.cpp:100] [control__parse_identify_results] firmware_version is: 4.18.1
[2024-08-20 12:21:55.301] [596] [HailoRT] [info] [core_op.cpp:156] [activate] Activating scdepthv3 took 0.948994 milliseconds. Note that the function is asynchronous and thus the network is not fully activated >
[2024-08-20 12:21:55.301] [596] [HailoRT] [info] [core_op.cpp:156] [activate] Activating scdepthv3 took 1.030993 milliseconds. Note that the function is asynchronous and thus the network is not fully activated >
[2024-08-20 12:22:05.305] [605] [HailoRT] [error] [async_stream_base.hpp:167] [cv_wait_for] Got HAILO_TIMEOUT while waiting for output stream buffer scdepthv3/conv31
[2024-08-20 12:22:05.305] [605] [HailoRT] [error] [vdevice_native_stream.cpp:334] [read_impl] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Failed read from stream (device: 0000:04:00.0)
[2024-08-20 12:22:05.313] [606] [HailoRT] [error] [async_stream_base.hpp:82] [cv_wait_for] Got HAILO_TIMEOUT while waiting for input stream buffer scdepthv3/input_layer1
[2024-08-20 12:22:05.313] [606] [HailoRT] [error] [vdevice_native_stream.cpp:145] [write_impl] CHECK_SUCCESS failed with status=HAILO_TIMEOUT(4) - Failed write to stream (device: 0000:04:00.0)
[2024-08-20 12:22:10.311] [596] [HailoRT] [info] [core_op.cpp:186] [deactivate] Deactivating took 0.027 ms
[2024-08-20 12:22:10.311] [596] [HailoRT] [info] [core_op.cpp:186] [deactivate] Deactivating took 0.151999 ms
In my Yocto project I use branch honister from meta-hailo repository. I am aware of the lack of support for the risc architecture, but maybe these errors comes from reasons other than architecture.
Thanks in advance