Insufficient CMA memory

[makro]

Hello everyone!

A year ago I tried to integrate Hailo-8 with a RISC-V based SoC platform. I encountered problems that I couldn’t solve at that time. They are described in the thread below:

Errors with HailoRT CLI commands on RISC-V platform

Hi everyone! I tried to run Hailo-8 on platform based on RISC-V architecture. I know it’s not officially supported, but I successfully implemented meta-hailo in my Yocto environment. All conditions from the subsection" Validating the Integration’s Success" are met. Hailo-8 device is visible using lspci and hailortcli scan commands. I’m also able to measure power consumption using hailortcli measure-power command. The problems start when I try to run .hef network or use benchmark on .hef fil…

I recently returned to the project because the hailo drivers were updated and I built a new Linux with a newer kernel for my platform.

This time I encountered a different problem. The situation is partly similar, because the only thing I can do with hailortcli is scan Hailo devices and measure power. However, trying to use the run or benchmark commands this time results in a message about insufficient CMA memory.

Log from terminal after running hailortcli run:

Running streaming inference (tiny_yolov4.hef):
Transform data: true
Type: auto
Quantized: true
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed to create desc list
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed configure vdevice from hef

Log from hailort.log file:

[2025-05-29 13:41:22.728] [314] [HailoRT] [info] [vdevice.cpp:535] [create] Creating vdevice with params: device_count: 1, scheduling_algorithm: NONE, multi_process_service: false
[2025-05-29 13:41:22.767] [314] [HailoRT] [info] [device.cpp:51] [Device] OS Version: Linux 6.6.51-linux4microchip+fpga-2024.09-g56d7964a57cd-dirty #1 SMP Wed Oct 9 11:00:43 UTC 2024 riscv64
[2025-05-29 13:41:22.770] [314] [HailoRT] [info] [control.cpp:113] [control__parse_identify_results] firmware_version is: 4.21.0
[2025-05-29 13:41:22.770] [314] [HailoRT] [info] [vdevice.cpp:681] [create] VDevice Infos: 0000:05:00.0
[2025-05-29 13:41:22.786] [314] [HailoRT] [error] [hailort_driver.cpp:1049] [descriptors_list_create] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed to create desc list
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager.cpp:461] [create_boundary_vdma_channel] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager_builder.cpp:1046] [create_boundary_channels] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager_builder.cpp:1333] [build] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:165] [create_configured_network_group] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:382] [create_networks_group_vector] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:144] [add_hef] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [device_internal.cpp:54] [configure] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdevice.cpp:938] [create_physical_core_op] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.789] [314] [HailoRT] [error] [vdevice.cpp:966] [create_vdevice_core_op] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.789] [314] [HailoRT] [error] [vdevice.cpp:748] [configure] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)

Logs from dmesg while booting device:

[ 11.081601] hailo_pci: loading out-of-tree module taints kernel.
[ 11.152092] hailo: Init module. driver version 4.21.0
[ 11.163867] hailo 0000:05:00.0: Probing on: 1e60:2864…
[ 11.169419] hailo 0000:05:00.0: Probing: Allocate memory for device extension, 13192
[ 11.177444] pci 0000:00:00.0: enabling device (0000 → 0003)
[ 11.183237] pci 0000:01:00.0: enabling device (0000 → 0003)
[ 11.189016] pci 0000:02:10.0: enabling device (0000 → 0002)
[ 11.194794] hailo 0000:05:00.0: enabling device (0000 → 0002)
[ 11.200742] hailo 0000:05:00.0: Probing: Device enabled
[ 11.206149] hailo 0000:05:00.0: Probing: mapped bar 0 - 000000005ca82420 16384
[ 11.213600] hailo 0000:05:00.0: Probing: mapped bar 2 - 000000000bec8697 4096
[ 11.220906] hailo 0000:05:00.0: Probing: mapped bar 4 - 000000000ba46afe 16384
[ 11.228293] hailo 0000:05:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[ 11.237089] hailo 0000:05:00.0: Probing: Enabled 64 bit dma
[ 11.242796] hailo 0000:05:00.0: Probing: Using userspace allocated vdma buffers
[ 11.250271] hailo 0000:05:00.0: Disabling ASPM L0s
[ 11.255242] hailo 0000:05:00.0: Successfully disabled ASPM L0s
[ 11.261604] hailo 0000:05:00.0: Writing file hailo/hailo8_fw.bin
[ 11.347759] hailo 0000:05:00.0: File hailo/hailo8_fw.bin written successfully
[ 11.355043] hailo 0000:05:00.0: Writing file hailo/hailo8_board_cfg.bin
[ 11.361915] hailo 0000:05:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 11.369809] hailo 0000:05:00.0: Writing file hailo/hailo8_fw_cfg.bin
[ 11.376419] hailo 0000:05:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 11.482433] hailo 0000:05:00.0: NNC Firmware loaded successfully
[ 11.488644] hailo 0000:05:00.0: FW loaded, took 227 ms
[ 11.507049] hailo 0000:05:00.0: Probing: Added board 1e60-2864, /dev/hailo0

Logs from dmesg after hailortcli run command:

[ 327.981620] hailo 0000:05:00.0: Failed to allocate descriptors list, desc_count 0x8000, buffer_size 0x80000, This failure means there is not a sufficient amount of CMA memory (contiguous physical memory), This usually is caused by lack of general system memory. Please check you have sufficient memory.
[ 327.981681] hailo 0000:05:00.0: failed to allocate descriptors buffer
[ 641.875131] hailo 0000:05:00.0: Failed to allocate descriptors list, desc_count 0x8000, buffer_size 0x80000, This failure means there is not a sufficient amount of CMA memory (contiguous physical memory), This usually is caused by lack of general system memory. Please check you have sufficient memory.
[ 641.875189] hailo 0000:05:00.0: failed to allocate descriptors buffer

To solve this problem I reserved 1 GB of CMA memory but it didn’t helped. I think that CMA memory is reserved correctly because it is also in logs from dmesg:

[ 0.000000] Reserved memory: created CMA memory pool at 0x0000001000000000, size 1024 MiB

I know that RISC-V is not officialy supported but this time error is clear. As for the platform, 1 GB is the maximum value to declare for CMA memory, but it seems to me that this should be sufficient for applications with Hailo, so the cause of the problem must be somewhere else.

Thanks in advance!