The inference program I wrote in C++ keeps returning the error “Ioctl HAILO_VDMA_LAUNCH_TRANSFER failed due to invalid address.” I suspect there might be a bug in the low-level driver, or is there a minimum required DMA size in the system configuration? I also get this error when running the official hailortcli command.
hailortcli run -t 36000 --measure-temp /home/admin/yolov6n.hef measure-stats --elem-fps --elem-queue-size
Running streaming inference (/home/admin/yolov6n.hef):
Transform data: true
Type: auto
Quantized: true
Network yolov6n/yolov6n: 27% | 9798098 | FPS: 983.06 | ETA: 07:13:53
[HailoRT] [error] Ioctl HAILO_VDMA_LAUNCH_TRANSFER failed due to invalid address
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6) - Failed launch transfer
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6) - Failed write to stream (device: 0001:11:00.0)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6) - HwWriteEl3yolov6n/input_layer1 (H2D) failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] Queue element PushQEl3yolov6n/input_layer1 run in thread function failed! status = HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_INVALID_OPERATION(6)
Welcome to the Hailo Community!
Can you confirm that the HailoRT CLI run failed after about 3 hours?
How about your own application? Does it fail immediately or after some time?
the HailoRT CLI run failed after about 2 hours 45 minutes。I got the time from the hailort.log. Our program runs for a while before it fails. If the inference speed drops after running for a while, this error does not occur. Although no temperature alarm was triggered, it seems that the drop in inference speed was caused by throttling. If the inference speed does not drop, that error occurs. Is there a way to reduce the inference speed from the start?
While the Hailo-8 does not require active cooling in most designs, it does require a passive thermal solution to dissipate the generated heat. In many designs, the Hailo-8 is thermally coupled to the product enclosure, which serves as a heat sink.
You can use the Hailo-8 Integration Tool to validate your thermal solution. Note: We have identified an issue with the thermal test models and are currently working on an updated version of the tool for the Developer Zone.
If you are using a Hailo-8 M.2 module from Hailo, you can create a user configuration that limits the device’s clock speed. Please refer to the following forum post for more information:
Hailo Community - Hailo-8 fixed systems clock speed option
However, I suspect your application is not running the YOLOv6n model at its maximum frame rate. In that case, limiting the clock speed is unlikely to have a significant impact on average power consumption and therefore chip temperature. I recommend addressing the thermal solution first.
For testing, you can use the hailortcli run2 command, which allows you to set the target frame rate for each model:
hailortcli run2 set-net yolov6n.hef --framerate 30
After startup, the maximum frame rate reaches 400fps with a Utilization of around 80%. When the NPU temperature is around 55 degrees, the frame rate drops to around 255fps and the Utilization increases to around 99%. At this point, it can run without issues for a whole night. 55 degrees should not be the critical temperature for NPU frequency reduction. If the frame rate does not drop to around 255fps, errors are prone to occur. Modifying the frequency through configuration files may not be supported by my device.
hailortcli fw-config read --output-file config.json
Executing on device: 0001:11:00.0
[HailoRT] [warning] Failed to examine user config, as it is not loaded or is not supported by the device.
[HailoRT] [error] Firmware control has failed. Major status: 0x40030033, Minor status: 0x401a0004
[HailoRT] [error] Firmware major status: CONTROL_PROTOCOL_STATUS_USER_CONFIG_EXAMINE_FAILED
[HailoRT] [error] Firmware minor status: FIRMWARE_CONFIGS_STATUS_USER_CONFIG_NOT_LOADED
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed to examine user config
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed reading user config from device
Failed to execute on device: 0001:11:00.0. status= HAILO_FW_CONTROL_FAILURE(18)
hailortcli fw-config write config.json
Executing on device: 0001:11:00.0
[HailoRT] [error] Firmware control has failed. Major status: 0x401a0002, Minor status: 0x401a0002
[HailoRT] [error] Firmware major status: FIRMWARE_CONFIGS_STATUS_USER_CONFIG_WRITE_NOT_ALLOWED
[HailoRT] [error] Firmware minor status: FIRMWARE_CONFIGS_STATUS_USER_CONFIG_WRITE_NOT_ALLOWED
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_FW_CONTROL_FAILURE(18) - Failed writing user firmware configuration to device
Failed to execute on device: 0001:11:00.0. status= HAILO_FW_CONTROL_FAILURE(18)
Yes, that is possible. As mentioned in the other post the EEPROM required for this feature is optional and some designs do not implement it.
That seems strange. Do you get an errors or warning from HailoRT about switching frequency?
Could the issue be in the PCIe link? Can you validate the PCIe link status before and after the event e.g. sudo lspci -vv ?