Insufficient CMA memory

General
, ,
1

Hello everyone!

A year ago I tried to integrate Hailo-8 with a RISC-V based SoC platform. I encountered problems that I couldn’t solve at that time. They are described in the thread below:

I recently returned to the project because the hailo drivers were updated and I built a new Linux with a newer kernel for my platform.

This time I encountered a different problem. The situation is partly similar, because the only thing I can do with hailortcli is scan Hailo devices and measure power. However, trying to use the run or benchmark commands this time results in a message about insufficient CMA memory.

Log from terminal after running hailortcli run:

Running streaming inference (tiny_yolov4.hef):
Transform data: true
Type: auto
Quantized: true
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed to create desc list
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed configure vdevice from hef

Log from hailort.log file:

[2025-05-29 13:41:22.728] [314] [HailoRT] [info] [vdevice.cpp:535] [create] Creating vdevice with params: device_count: 1, scheduling_algorithm: NONE, multi_process_service: false
[2025-05-29 13:41:22.767] [314] [HailoRT] [info] [device.cpp:51] [Device] OS Version: Linux 6.6.51-linux4microchip+fpga-2024.09-g56d7964a57cd-dirty #1 SMP Wed Oct 9 11:00:43 UTC 2024 riscv64
[2025-05-29 13:41:22.770] [314] [HailoRT] [info] [control.cpp:113] [control__parse_identify_results] firmware_version is: 4.21.0
[2025-05-29 13:41:22.770] [314] [HailoRT] [info] [vdevice.cpp:681] [create] VDevice Infos: 0000:05:00.0
[2025-05-29 13:41:22.786] [314] [HailoRT] [error] [hailort_driver.cpp:1049] [descriptors_list_create] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81) - Failed to create desc list
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager.cpp:461] [create_boundary_vdma_channel] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager_builder.cpp:1046] [create_boundary_channels] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.787] [314] [HailoRT] [error] [resource_manager_builder.cpp:1333] [build] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:165] [create_configured_network_group] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:382] [create_networks_group_vector] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdma_device.cpp:144] [add_hef] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [device_internal.cpp:54] [configure] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.788] [314] [HailoRT] [error] [vdevice.cpp:938] [create_physical_core_op] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.789] [314] [HailoRT] [error] [vdevice.cpp:966] [create_vdevice_core_op] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)
[2025-05-29 13:41:22.789] [314] [HailoRT] [error] [vdevice.cpp:748] [configure] CHECK_SUCCESS failed with status=HAILO_OUT_OF_HOST_CMA_MEMORY(81)

Logs from dmesg while booting device:

[ 11.081601] hailo_pci: loading out-of-tree module taints kernel.
[ 11.152092] hailo: Init module. driver version 4.21.0
[ 11.163867] hailo 0000:05:00.0: Probing on: 1e60:2864…
[ 11.169419] hailo 0000:05:00.0: Probing: Allocate memory for device extension, 13192
[ 11.177444] pci 0000:00:00.0: enabling device (0000 → 0003)
[ 11.183237] pci 0000:01:00.0: enabling device (0000 → 0003)
[ 11.189016] pci 0000:02:10.0: enabling device (0000 → 0002)
[ 11.194794] hailo 0000:05:00.0: enabling device (0000 → 0002)
[ 11.200742] hailo 0000:05:00.0: Probing: Device enabled
[ 11.206149] hailo 0000:05:00.0: Probing: mapped bar 0 - 000000005ca82420 16384
[ 11.213600] hailo 0000:05:00.0: Probing: mapped bar 2 - 000000000bec8697 4096
[ 11.220906] hailo 0000:05:00.0: Probing: mapped bar 4 - 000000000ba46afe 16384
[ 11.228293] hailo 0000:05:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[ 11.237089] hailo 0000:05:00.0: Probing: Enabled 64 bit dma
[ 11.242796] hailo 0000:05:00.0: Probing: Using userspace allocated vdma buffers
[ 11.250271] hailo 0000:05:00.0: Disabling ASPM L0s
[ 11.255242] hailo 0000:05:00.0: Successfully disabled ASPM L0s
[ 11.261604] hailo 0000:05:00.0: Writing file hailo/hailo8_fw.bin
[ 11.347759] hailo 0000:05:00.0: File hailo/hailo8_fw.bin written successfully
[ 11.355043] hailo 0000:05:00.0: Writing file hailo/hailo8_board_cfg.bin
[ 11.361915] hailo 0000:05:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 11.369809] hailo 0000:05:00.0: Writing file hailo/hailo8_fw_cfg.bin
[ 11.376419] hailo 0000:05:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 11.482433] hailo 0000:05:00.0: NNC Firmware loaded successfully
[ 11.488644] hailo 0000:05:00.0: FW loaded, took 227 ms
[ 11.507049] hailo 0000:05:00.0: Probing: Added board 1e60-2864, /dev/hailo0

Logs from dmesg after hailortcli run command:

[ 327.981620] hailo 0000:05:00.0: Failed to allocate descriptors list, desc_count 0x8000, buffer_size 0x80000, This failure means there is not a sufficient amount of CMA memory (contiguous physical memory), This usually is caused by lack of general system memory. Please check you have sufficient memory.
[ 327.981681] hailo 0000:05:00.0: failed to allocate descriptors buffer
[ 641.875131] hailo 0000:05:00.0: Failed to allocate descriptors list, desc_count 0x8000, buffer_size 0x80000, This failure means there is not a sufficient amount of CMA memory (contiguous physical memory), This usually is caused by lack of general system memory. Please check you have sufficient memory.
[ 641.875189] hailo 0000:05:00.0: failed to allocate descriptors buffer

To solve this problem I reserved 1 GB of CMA memory but it didn’t helped. I think that CMA memory is reserved correctly because it is also in logs from dmesg:

[ 0.000000] Reserved memory: created CMA memory pool at 0x0000001000000000, size 1024 MiB

I know that RISC-V is not officialy supported but this time error is clear. As for the platform, 1 GB is the maximum value to declare for CMA memory, but it seems to me that this should be sufficient for applications with Hailo, so the cause of the problem must be somewhere else.

Thanks in advance!

2

Hey @makro,

Looking at your logs:

  • Hailo-8 shows up fine with lspci and hailortcli scan
  • Power measurement is working: hailortcli measure-power
  • CMA pool looks like it’s reserved properly:
    [ 0.000000] Reserved memory: created CMA memory pool at 0x0000001000000000, size 1024 MiB

So the PCIe device is working and firmware loaded correctly.

That error is pretty clear:

Failed to allocate descriptors list, desc_count 0x8000, buffer_size 0x80000
This failure means there is not a sufficient amount of CMA memory (contiguous physical memory)

What I’d try:

1. Check CMA Region Placement
The reserved memory region can’t conflict with other stuff. Check your Device Tree .dts file:

reserved-memory {
    #address-cells = <2>;
    #size-cells = <2>;
    ranges;
    cma@100000000 {
        compatible = "shared-dma-pool";
        reusable;
        size = <0x0 0x40000000>; // 1GB
        alignment = <0x0 0x1000>;
        no-map;
    };
};

2. Test with a simple HEF like MobileNet first

3. Check CMA status before running with hailortcli run:

cat /proc/meminfo | grep Cma
dmesg | grep -i cma

Make sure your kernel has:

CONFIG_CMA=y
CONFIG_DMA_CMA=y

Let me know what those commands show!

3

Hi omria!

Thank you for response.

If you need more information, this is how I declared CMA memory in my device tree:

ddrc_cache_lo: memory@80000000 {
		device_type = "memory";
		reg = <0x0 0x80000000 0x0 0x30000000>;
	};

	ddrc_cache_hi: memory@1000000000 {
		device_type = "memory";
		reg = <0x10 0x0 0x0 0x40000000>;
	};

	leds {
		compatible = "gpio-leds";
		led-0 {
			gpios = <&gpio1 23 1>;
			default-state = "on";
			linux,default-trigger = "heartbeat";
		};
	};
	
	reserved-memory {
	#address-cells = <2>;
	#size-cells = <2>;
	ranges;

	linux,cma@1000000000 {
		compatible = "shared-dma-pool";
		reusable;
		reg = <0x10 0x0 0x0 0x40000000>; // 1 GiB at 0x1000000000
		alignment = <0x0 0x1000>;        // 4 KiB
		linux,cma-default;
	};

For tests, I use tiny_yolov4 HEF file.

These are the results from using the commands given above

CmaTotal:        1048576 kB
CmaFree:          950848 kB

[    0.000000] Reserved memory: created CMA memory pool at 0x0000001000000000, size 1024 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma@1000000000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000001000000000..0x000000103fffffff (1048576 KiB) map reusable linux,cma@1000000000
[    0.000000] Memory: 654620K/1835008K available (7120K kernel code, 4833K rwdata, 4096K rodata, 2161K init, 385K bss, 131812K reserved, 1048576K cma-reserved)

I am also sure that mentioned kernel configs are enabled

1 Like