ERROR - hailo_tools_runner.py:579 - Failed to produce compiled graph, ERROR - hailo_tools_runner.py:231 - BackendAllocatorException: Compilation failed with unexpected crash

General
1

I am trying to convert an ONNX to HEF using Hailo Dataflow Compiler 3.29. I have ensured the correct versions of HailoRT, Hailo Model Zoo, etc. for the necessary DFC version. My ONNX file was created using the Google Colab file provided in (Raspberry Pi AI Kit: Training Models with Google Colab). However, I am getting the below error:

2025-06-24 01:47:19,182 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-06-24 01:47:19,563 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.38)
2025-06-24 01:47:20,967 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:01.79)
2025-06-24 01:47:22,930 - INFO - onnx_translator.py:2482 - NMS structure of yolov8 (or equivalent architecture) was detected.
2025-06-24 01:47:22,930 - INFO - onnx_translator.py:2484 - In order to use HailoRT post-processing capabilities, these end node names should be used: /model.22/cv2.0/cv2.0.2/Conv /model.22/cv3.0/cv3.0.2/Conv /model.22/cv2.1/cv2.1.2/Conv /model.22/cv3.1/cv3.1.2/Conv /model.22/cv2.2/cv2.2.2/Conv /model.22/cv3.2/cv3.2.2/Conv.
2025-06-24 01:47:23,060 - INFO - parser.py:373 - Start nodes mapped from original model: ‘images’: ‘yolov8s/input_layer1’.
2025-06-24 01:47:23,060 - INFO - parser.py:374 - End nodes mapped from original model: ‘/model.22/cv2.0/cv2.0.2/Conv’, ‘/model.22/cv3.0/cv3.0.2/Conv’, ‘/model.22/cv2.1/cv2.1.2/Conv’, ‘/model.22/cv3.1/cv3.1.2/Conv’, ‘/model.22/cv2.2/cv2.2.2/Conv’, ‘/model.22/cv3.2/cv3.2.2/Conv’.
2025-06-24 01:47:23,439 - INFO - parser.py:288 - Translation completed on ONNX model yolov8s (completion time: 00:00:04.26)
2025-06-24 01:47:26,146 - INFO - client_runner.py:1831 - Saved HAR to: /root/yolov8s.har
2025-06-24 01:47:26,983 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /root/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-06-24 01:47:27,059 - INFO - client_runner.py:479 - Loading model script commands to yolov8s from string
2025-06-24 01:47:28,251 - IMPORTANT - sdk_backend.py:1009 - Starting Model Optimization
2025-06-24 01:49:47,454 - IMPORTANT - sdk_backend.py:1050 - Model Optimization is done
2025-06-24 01:49:49,113 - INFO - client_runner.py:1831 - Saved HAR to: /root/yolov8s.har
2025-06-24 01:49:49,117 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /root/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-06-24 01:49:49,185 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-06-24 01:49:49,186 - INFO - client_runner.py:479 - Appending model script commands to yolov8s from string
2025-06-24 01:49:49,197 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-06-24 07:47:27,033 - ERROR - hailo_tools_runner.py:579 - Failed to produce compiled graph
2025-06-24 07:47:27,230 - ERROR - hailo_tools_runner.py:231 - BackendAllocatorException: Compilation failed with unexpected crash

Can you please help me out with this? I have converted an ONNX to HEF with this exact same guide before multiple times on the same device, never had a problem. Thanks!

2

Update: I have used the google colab from the above mentioned tutorial (cytron), and the one from Roboflow too, which redirects once you copy a model. Does the Ultralytics version matter too in this case? I am getting the same result for both. Your help would be grateful, thanks!

3

@omria

could you please help with this? I have tried everything possible.

4

What is the command that you’ve used? What is the input onnx?

5

Command used:

hailomz compile yolov8s --ckpt=best_29_06_2025_converted.onnx --hw-arch hailo8l --calib-path train/images --classes 3 --performance

Latest error log file and onnx:

Platform:
Hailo DFC 3.29
Python 3.10
HailoRT 4.19
Hailo Model Zoo: 2.13

Steps followed: Cytron tutorials (Part 1 and 2):

I’d be glad for your help, thanks. I have done this on the same device multiple times and using the same tutorial as a reference, Prior to Dec 2024.

6

Just making sure, are you using Windows Subsystem for Linux?

I recreated an environment with

(hailodfc) wsl@TYANAPRIL232024:~$ pip list | grep hailo
hailo-dataflow-compiler      3.29.0
hailo-model-zoo              2.13.0         /home/wsl/hailo_model_zoo-2.13
hailort                      4.19.0

and I saw these additional lines in the terminal before the [error] Failed to produce compiled graph if induced manually with a CTRL+C:

[info] ParsedPerformanceParam command, setting optimization_level(max=2)
[info] Appending model script commands to yolov8s from string
[info] ParsedPerformanceParam command, setting optimization_level(max=2)
[info] Loading network parameters
[info] Starting Hailo allocation and compilation flow
[info] Adding an output layer after conv41
<...>

Here’s my idea:

Your log indicates that your hailo_model_zoo is at /root/: /root/hailo_model_zoo/hailo_model_zoo/cfg/alls/generic/yolov8s.alls, but WSL cannot access /root/.
Try:

  1. Getting hailo_model_zoo from Release v2.13: Merge pull request #128 from hailo-ai/update-to-version-2.13 · hailo-ai/hailo_model_zoo · GitHub
  2. Unzipping into a location like ~/hailo_model_zoo-2.13
  3. pip install -e in hailo_model_zoo-2.13 (the same directory as setup.py)
  4. Rerunning hailomz compile yolov8s --ckpt=best_29_06_2025_converted.onnx --hw-arch hailo8l --calib-path train/images --classes 3 --performance

You might need to move all files relevant to the hailomz compile command out of /root, such as the .onnx and train/images folder.

7

A first trial would be to omit the --performance flag, which pushes hard on the compiler.

8

Hi,

Yes, I am using WSL2. Have converted an ONNX to HEF with Hailo DFC 3.29 multiple times earlier in the same setup and venv.

Regarding the root folder, I did exactly what you said. You can see my new error file in the drive link uploaded previously (here)

Still got the same error.

I did the exact same commands with the latest TAPPAS, Hailo MZ, etc. and got a successful HEF (same folder as the one in the error folder - \wsl$\Ubuntu-22.04\home\atharva). I got a successful HEF.

But alas, my entire project is based on Hailo DFC 3.29, HailoRT 4.19, etc.

9

I have done it with the --performance flag multiple times earlier. I will try it without the flag, and report to you. Thanks.

10

I was able to log a hailo_sdk.client.log with an exit point very similar to your log. Here’s mine:

2025-07-01 13:41:25,301 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-07-01 13:41:25,528 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.23)
2025-07-01 13:41:26,081 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.78)
2025-07-01 13:44:48,227 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-07-01 13:44:48,412 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.18)
2025-07-01 13:44:48,939 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.71)
2025-07-01 13:45:10,429 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-07-01 13:45:10,604 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.17)
2025-07-01 13:45:11,092 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.66)
2025-07-01 13:45:49,196 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-07-01 13:45:49,364 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.17)
2025-07-01 13:45:49,984 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.79)
2025-07-01 13:45:50,804 - INFO - onnx_translator.py:2482 - NMS structure of yolov8 (or equivalent architecture) was detected.
2025-07-01 13:45:50,804 - INFO - onnx_translator.py:2484 - In order to use HailoRT post-processing capabilities, these end node names should be used: /model.22/cv2.0/cv2.0.2/Conv /model.22/cv3.0/cv3.0.2/Conv /model.22/cv2.1/cv2.1.2/Conv /model.22/cv3.1/cv3.1.2/Conv /model.22/cv2.2/cv2.2.2/Conv /model.22/cv3.2/cv3.2.2/Conv.
2025-07-01 13:45:50,881 - INFO - parser.py:373 - Start nodes mapped from original model: 'images': 'yolov8s/input_layer1'.
2025-07-01 13:45:50,881 - INFO - parser.py:374 - End nodes mapped from original model: '/model.22/cv2.0/cv2.0.2/Conv', '/model.22/cv3.0/cv3.0.2/Conv', '/model.22/cv2.1/cv2.1.2/Conv', '/model.22/cv3.1/cv3.1.2/Conv', '/model.22/cv2.2/cv2.2.2/Conv', '/model.22/cv3.2/cv3.2.2/Conv'.
2025-07-01 13:45:50,957 - INFO - parser.py:288 - Translation completed on ONNX model yolov8s (completion time: 00:00:01.76)
2025-07-01 13:45:52,218 - INFO - client_runner.py:1831 - Saved HAR to: /home/wsl/yolov8s.har
2025-07-01 13:45:52,811 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /home/wsl/hailo_model_zoo-2.13/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-07-01 13:45:52,856 - INFO - client_runner.py:479 - Loading model script commands to yolov8s from string
2025-07-01 13:45:53,919 - IMPORTANT - sdk_backend.py:1009 - Starting Model Optimization
2025-07-01 13:47:03,698 - IMPORTANT - sdk_backend.py:1050 - Model Optimization is done
2025-07-01 13:47:04,759 - INFO - client_runner.py:1831 - Saved HAR to: /home/wsl/yolov8s.har
2025-07-01 13:47:04,762 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /home/wsl/hailo_model_zoo-2.13/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-07-01 13:47:04,824 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-07-01 13:47:04,825 - INFO - client_runner.py:479 - Appending model script commands to yolov8s from string
2025-07-01 13:47:04,835 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-07-01 16:43:01,918 - INFO - client_runner.py:1831 - Saved HAR to: /home/wsl/yolov8s.har
2025-07-01 17:02:34,806 - INFO - parser.py:158 - Translation started on ONNX model yolov8s
2025-07-01 17:02:34,965 - INFO - parser.py:193 - Restored ONNX model yolov8s (completion time: 00:00:00.16)
2025-07-01 17:02:35,634 - INFO - parser.py:216 - Extracted ONNXRuntime meta-data for Hailo model (completion time: 00:00:00.83)
2025-07-01 17:02:36,579 - INFO - onnx_translator.py:2482 - NMS structure of yolov8 (or equivalent architecture) was detected.
2025-07-01 17:02:36,579 - INFO - onnx_translator.py:2484 - In order to use HailoRT post-processing capabilities, these end node names should be used: /model.22/cv2.0/cv2.0.2/Conv /model.22/cv3.0/cv3.0.2/Conv /model.22/cv2.1/cv2.1.2/Conv /model.22/cv3.1/cv3.1.2/Conv /model.22/cv2.2/cv2.2.2/Conv /model.22/cv3.2/cv3.2.2/Conv.
2025-07-01 17:02:36,694 - INFO - parser.py:373 - Start nodes mapped from original model: 'images': 'yolov8s/input_layer1'.
2025-07-01 17:02:36,694 - INFO - parser.py:374 - End nodes mapped from original model: '/model.22/cv2.0/cv2.0.2/Conv', '/model.22/cv3.0/cv3.0.2/Conv', '/model.22/cv2.1/cv2.1.2/Conv', '/model.22/cv3.1/cv3.1.2/Conv', '/model.22/cv2.2/cv2.2.2/Conv', '/model.22/cv3.2/cv3.2.2/Conv'.
2025-07-01 17:02:36,967 - INFO - parser.py:288 - Translation completed on ONNX model yolov8s (completion time: 00:00:02.16)
2025-07-01 17:02:38,331 - INFO - client_runner.py:1831 - Saved HAR to: /root/yolov8s.har
2025-07-01 17:02:38,996 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /home/wsl/hailo_model_zoo-2.13/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-07-01 17:02:39,043 - INFO - client_runner.py:479 - Loading model script commands to yolov8s from string
2025-07-01 17:02:42,257 - IMPORTANT - sdk_backend.py:1009 - Starting Model Optimization
2025-07-01 17:04:00,876 - IMPORTANT - sdk_backend.py:1050 - Model Optimization is done
2025-07-01 17:04:02,056 - INFO - client_runner.py:1831 - Saved HAR to: /root/yolov8s.har
2025-07-01 17:04:02,061 - INFO - client_runner.py:476 - Loading model script commands to yolov8s from /home/wsl/hailo_model_zoo-2.13/hailo_model_zoo/cfg/alls/generic/yolov8s.alls
2025-07-01 17:04:02,109 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-07-01 17:04:02,110 - INFO - client_runner.py:479 - Appending model script commands to yolov8s from string
2025-07-01 17:04:02,118 - INFO - commands.py:4064 - ParsedPerformanceParam command, setting optimization_level(max=2)
2025-07-01 17:04:02,190 - ERROR - hailo_tools_runner.py:231 - PermissionError: [Errno 13] Permission denied: '/tmp/reflection.alls'

Notice how the very last line is different:

2025-07-01 17:04:02,190 - ERROR - hailo_tools_runner.py:231 - PermissionError: [Errno 13] Permission denied: ‘/tmp/reflection.alls’

To encounter this error, I did sudo su, activated the venv built using wsl, then ran the hailomz compile. This error persisted until I did a rm -f /tmp/reflection.alls. After removing that temp file, I was successfully able to compile the HEF.
As such, try running rm -f /tmp/reflection.alls and then your build command. Maybe even wipe out the entile tmp folder.

I received the permission error because my temp directory’s reflection.alls was created by my wsl account in wsl, and so root didn’t have permissions to deal with that file.

Environment was copied into /root/ from my home directory without further modifications.:

(hailodfc) root@TYANAPRIL232024:~# pip list | grep hailo
hailo-dataflow-compiler      3.29.0
hailo-model-zoo              2.13.0         /home/wsl/hailo_model_zoo-2.13
hailort                      4.19.0
11

Hi,

@Raytao_Xia:

I followed your instructions but am still encountering the same error. I’ve attached all relevant logs for your reference here.

A few clarifications that might be relevant:

  • The system is running Ubuntu 22.04 (previously 22.04 was used as well).

  • numpy and numba versions were explicitly downgraded after installing HailoRT (via both .whl and .deb) to resolve pip dependency conflicts:

    pip install numba==0.58.1 numpy==1.23.3

  • The HailoRT service does not appear to be activated when checking via pip list | grep hailo, despite installing both the .deb and .whl packages and confirming activation when prompted:

    Screenshot (382)
    Screenshot (382)919×242 79.5 KB

@Nadav:

I ran the command without the --performance flag and received the HEF in approximately 30 minutes. Would this HEF be as accurate and reliable as those previously generated after over 6 hours of compilation with the --performance flag enabled?

Thanks!

12

Weird. Glad you have a solution by simply taking off --performance though.

13

Yes. Accuracy will be determined during optimization.

During compilation the Hailo Dataflow Compiler will allocate the quantized network on the Hailo device. With the --performance flag it will try harder to use more resources so the network can run faster. This can fail and the compiler will try again. Without the flag the compiler will play it safe and make sure it can allocate the network successfully the first time.

Please check the HailoRT User Guide section 6.12. Multi-Process Service on how to enable the service.

14

Hi,
Yes, the --performance flag is a pure compiler flag, it doesn’t affect the accuracy. It applies “best-effort” heuristics trying to push the compiler to get highest FPS, sometimes this results in conditions that do not converge. I like the fact that in many cases you were able to use the performance flag without any issue.

By the way, you can always use the hailomz eval command in order to evaluate the accuracy of your optimized or compiled model.