Hailo-10H HAILO_COMMUNICATION_CLOSED(62) on Jetson AGX Orin with HailoRT 5.3.0

Hi,

I’m experiencing HAILO_COMMUNICATION_CLOSED(62) error with Hailo-10H on
Jetson AGX Orin. All software components are correctly installed and the
firmware loads successfully, but hailortcli fw-control identify fails.

Environment

  • Hardware: NVIDIA Jetson AGX Orin Developer Kit (64GB)
  • Hailo device: Hailo-10H (M.2, via PCIe expansion board)
  • OS: Ubuntu 22.04.5 LTS
  • Kernel: 5.15.185-tegra (aarch64)
  • JetPack: 6.2.2 (L4T R36.5.0)
  • CUDA: 12.6
  • HailoRT: 5.3.0
  • Driver: hailo1x_pci 5.3.0 (DKMS built)

What works

PCIe device recognized:
$ lspci | grep -i hailo
0005:01:00.0 Co-processor: Hailo Technologies Ltd. Device 45c4 (rev 01)

Firmware loads successfully (~1.7 seconds):
[10.346632] hailo1x 0005:01:00.0: SOC Firmware Batch loaded successfully
[10.346640] hailo1x 0005:01:00.0: Firmware loaded in 1724 ms
[10.363675] hailo1x 0005:01:00.0: Device created at /dev/h1x-0

Device files created:
/dev/h1x-0 (crw-rw-rw-)
/dev/hailo0 → /dev/h1x-0 (symlink)

hailortcli scan works:
$ hailortcli scan
Hailo Devices:
[-] Device: pci/0005:01:00.0

What fails

$ hailortcli fw-control identify
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62) - Failed to set default notification callbacks for Device HRPC-client
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)

Firmware boot sequence (from dmesg)

The firmware loads the following files in sequence:

  1. customer_certificate.bin
  2. scu_fw.bin
  3. u-boot-0.dtb.signed
  4. u-boot-spl.bin
  5. fitImage (11MB Linux kernel image)
  6. image-fs (100MB Linux root filesystem)

This looks like a full Linux boot sequence rather than NPU-only mode.
Is this expected for Hailo-10H? Or should there be a different firmware
for NPU-only mode?

Attempted fixes (all failed)

  1. Module reload (modprobe -r / modprobe hailo1x_pci)
  2. force_hailo_pcie_boot_mode_over_bars=1
  3. force_hailo10h_legacy_mode=1
  4. Cold boot (DC adapter disconnected for 1+ minute)
  5. USB virtual ethernet activation (l4tbr0, usb0, usb1) - NO-CARRIER

Network interfaces (no Hailo Linux connection found)

usb0: <NO-CARRIER,BROADCAST,MULTICAST,UP> master l4tbr0 state DOWN
usb1: <NO-CARRIER,BROADCAST,MULTICAST,UP> master l4tbr0 state DOWN

ping 192.168.55.2 → 100% packet loss
ping 192.168.7.2 → 100% packet loss

Driver info

$ modinfo hailo1x_pci
filename: /lib/modules/5.15.185-tegra/updates/dkms/hailo1x_pci.ko
version: 5.3.0
vermagic: 5.15.185-tegra SMP preempt mod_unload modversions aarch64

Questions

  1. Is the firmware boot sequence (fitImage + image-fs) expected for
    Hailo-10H in NPU mode, or is this a Linux/AI Vision Module mode?
  2. If this is Linux mode, how do I switch to NPU mode for use with
    hailortcli and HailoRT API?
  3. Are there specific configurations required for Hailo-10H on
    Jetson AGX Orin with JetPack 6.2.2?

I found a similar case where downgrading HailoRT version resolved the issue (4.19→4.18 on Windows). Is there a known compatibility issue between HailoRT 5.3.0 and Hailo-10H firmware on Jetson platforms?

Any help would be greatly appreciated. Thank you.

Hi @Junyong_Shin,

  1. Is fitImage + image-fs expected? Yes - Hailo-10H is a SoC that boots an internal Linux running hailort_server.
  2. How to switch to NPU mode? You’re already in it - there is no separate NPU-only mode, the internal Linux manages the NPU via RPC over PCIe.
  3. Specific configs for Jetson AGX Orin + JetPack 6.2.2? No Jetson-specific docs exist, but load the driver with no_power_mode=Y to prevent D3 sleep (regarding “device visible but communication closed” please see here: Hailortcli fw-control identify: error 6).
  4. Version downgrade applicable? No - Hailo-10H requires HailoRT 5.x; the 4.x line is for Hailo-8/8L only and cannot be used.

Thanks,

Hi Michael, Thank you for the suggestions. I tried all approaches but errors persist. Here are the detailed results:

**1. no_power_mode=Y**
```bash
sudo modprobe -r hailo1x_pci
sudo modprobe hailo1x_pci no_power_mode=Y
sleep 10
hailortcli fw-control identify
```
Result: HAILO_COMMUNICATION_CLOSED(62) — no change

**2. Force D0 power state**
```bash
echo on | sudo tee /sys/bus/pci/devices/0005:01:00.0/power/control
```
Result: HAILO_COMMUNICATION_CLOSED(62) — no change

**3. PCIe reset**
```bash
echo 1 | sudo tee /sys/bus/pci/devices/0005:01:00.0/reset
```
Result: Error changed to HAILO_DRIVER_OPERATION_FAILED(36)

Failed to open device file /dev/h1x-0 with error 6 Failed to connect to server


**4. Cold boot (full DC power disconnect, 1+ minute)**
After PCIe reset (error 36 state), disconnected DC adapter completely, 
waited 1 minute, reconnected and booted.
Result: Reverted back to HAILO_COMMUNICATION_CLOSED(62)

**Key Observations**
- Firmware always loads successfully:

SOC Firmware Batch loaded successfully Firmware loaded in 1724 ms Device created at /dev/h1x-0

- Error 36 "Failed to connect to server" only appears after PCIe reset, 
  before cold boot
- Cold boot always reverts to error 62
- hailort_server inside Hailo-10H appears to not start or be unreachable

Questions

1. Is there any way to verify if hailort_server is actually running inside Hailo-10H after firmware boot?

2. Are there any methods to extract logs from Hailo-10H internal Linux?

3. Could this be a hardware issue with this specific Hailo-10H unit?

Thank you.

Best regards, Junyong

Hi @Junyong_Shin,

  1. Verifying hailort_server inside Hailo-10H - The fact that hailortcli scan finds the device but fw-control identify returns HAILO_COMMUNICATION_CLOSED tells us the PCIe link is up but the RPC channel isn’t establishing - which could be the server not reaching ready state, or something blocking the shared-memory communication path between host and device.
  2. Extracting logs from Hailo-10H internal Linux - Not possible through any user-facing interface.
  3. Hardware defect - Unlikely given that PCIe enumeration, firmware loading, and device file creation all succeed. The link is healthy.

Can you please run:
sudo dmesg | grep -iE "smmu|iommu|fault|arm-smmu"
This might be useful to further diagnose the issue.

Thanks,

Hi Michael,

Here are the dmesg results:

Key Hailo-related entries:
[9.455374] hailo1x 0005:01:00.0: Adding to iommu group 6
[9.456244] hailo1x 0005:01:00.0: Probing: Using specialized dma_ops=iommu_dma_ops

No SMMU faults or IOMMU errors detected.
ARM-SMMU (SMMUv2) is present but no fault entries appear for the Hailo device.

Full output:
[ 0.000822] pid_max: default: 32768 minimum: 301
[ 0.180114] iommu: Default domain type: Translated
[ 0.180120] iommu: DMA domain TLB invalidation policy: strict mode
[ 0.234973] PCI: CLS 0 bytes, default 64
[ 0.289021] arm-smmu 8000000.iommu: probing hardware configuration…
[ 0.289025] arm-smmu 8000000.iommu: SMMUv2 with:
[ 0.289032] arm-smmu 8000000.iommu: stage 1 translation
[ 0.289035] arm-smmu 8000000.iommu: stage 2 translation
[ 0.289037] arm-smmu 8000000.iommu: nested translation
[ 0.289042] arm-smmu 8000000.iommu: stream matching with 128 register groups
[ 0.289046] arm-smmu 8000000.iommu: 128 context banks (0 stage-2 only)
[ 0.289059] arm-smmu 8000000.iommu: Supported page sizes: 0x61311000
[ 0.289063] arm-smmu 8000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.289065] arm-smmu 8000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.290050] arm-smmu 10000000.iommu: probing hardware configuration…
[ 0.290054] arm-smmu 10000000.iommu: SMMUv2 with:
[ 0.290056] arm-smmu 10000000.iommu: stage 1 translation
[ 0.290056] arm-smmu 10000000.iommu: stage 2 translation
[ 0.290057] arm-smmu 10000000.iommu: nested translation
[ 0.290060] arm-smmu 10000000.iommu: stream matching with 128 register groups
[ 0.290063] arm-smmu 10000000.iommu: 128 context banks (0 stage-2 only)
[ 0.290069] arm-smmu 10000000.iommu: Supported page sizes: 0x61311000
[ 0.290070] arm-smmu 10000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.290071] arm-smmu 10000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.290410] arm-smmu 12000000.iommu: probing hardware configuration…
[ 0.290414] arm-smmu 12000000.iommu: SMMUv2 with:
[ 0.290417] arm-smmu 12000000.iommu: stage 1 translation
[ 0.290417] arm-smmu 12000000.iommu: stage 2 translation
[ 0.290418] arm-smmu 12000000.iommu: nested translation
[ 0.290421] arm-smmu 12000000.iommu: stream matching with 128 register groups
[ 0.290424] arm-smmu 12000000.iommu: 128 context banks (0 stage-2 only)
[ 0.290429] arm-smmu 12000000.iommu: Supported page sizes: 0x61311000
[ 0.290430] arm-smmu 12000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.290432] arm-smmu 12000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.323532] tegra-bpmp bpmp: Adding to iommu group 0
[ 2.382951] tegra-gpcdma 2600000.dma-controller: Adding to iommu group 1
[ 2.391065] tegra-i2c 3160000.i2c: Adding to iommu group 1
[ 2.395293] tegra-i2c 3180000.i2c: Adding to iommu group 1
[ 2.396776] tegra-i2c 3190000.i2c: Adding to iommu group 1
[ 2.398127] tegra-i2c 31b0000.i2c: Adding to iommu group 1
[ 2.399435] tegra-i2c 31c0000.i2c: Adding to iommu group 1
[ 2.400749] tegra-i2c 31e0000.i2c: Adding to iommu group 1
[ 2.402359] tegra-i2c c240000.i2c: Adding to iommu group 1
[ 2.404128] tegra-i2c c250000.i2c: Adding to iommu group 1
[ 2.432350] sdhci-tegra 3460000.mmc: Adding to iommu group 2
[ 2.433408] sdhci-tegra 3400000.mmc: Adding to iommu group 3
[ 5.768299] tegra194-pcie 14100000.pcie: Adding to iommu group 4
[ 5.904213] pcieport 0001:00:00.0: Adding to iommu group 4
[ 5.905416] tegra194-pcie 14160000.pcie: Adding to iommu group 5
[ 6.043978] pcieport 0004:00:00.0: Adding to iommu group 5
[ 6.044317] nvme 0004:01:00.0: Adding to iommu group 5
[ 6.045070] tegra194-pcie 141a0000.pcie: Adding to iommu group 6
[ 6.050561] nvme nvme0: 12/0/0 default/read/poll queues
[ 6.280467] pcieport 0005:00:00.0: Adding to iommu group 6
[ 6.282452] tegra-xusb 3610000.usb: Adding to iommu group 7
[ 6.298718] tegra-xudc 3550000.usb: Adding to iommu group 8
[ 7.382965] systemd[1]: systemd 249.11-0ubuntu3.20 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 7.803968] systemd[1]: Queued start job for default target Graphical Interface.
[ 8.140481] tegra-host1x 13e00000.host1x: Adding to iommu group 9
[ 8.141621] host1x-context host1x-ctx.0: Adding to iommu group 10
[ 8.141761] host1x-context host1x-ctx.1: Adding to iommu group 11
[ 8.141876] host1x-context host1x-ctx.2: Adding to iommu group 12
[ 8.141992] host1x-context host1x-ctx.3: Adding to iommu group 13
[ 8.142118] host1x-context host1x-ctx.4: Adding to iommu group 14
[ 8.142231] host1x-context host1x-ctx.5: Adding to iommu group 15
[ 8.142347] host1x-context host1x-ctx.6: Adding to iommu group 16
[ 8.142474] host1x-context host1x-ctx.7: Adding to iommu group 17
[ 8.142593] host1x-context host1x-ctx.8: Adding to iommu group 18
[ 8.142703] host1x-context host1x-ctx.9: Adding to iommu group 19
[ 8.142817] host1x-context host1x-ctx.10: Adding to iommu group 20
[ 8.142927] host1x-context host1x-ctx.11: Adding to iommu group 21
[ 8.143027] host1x-context host1x-ctx.12: Adding to iommu group 22
[ 8.143139] host1x-context host1x-ctx.13: Adding to iommu group 23
[ 8.143246] host1x-context host1x-ctx.14: Adding to iommu group 24
[ 8.143359] host1x-context host1x-ctx.15: Adding to iommu group 25
[ 8.659107] tegra-se 15820000.crypto: Adding to iommu group 26
[ 8.670180] tegra-soc-hwpm f100000.hwpm: Adding to iommu group 28
[ 8.670226] tsec 15500000.tsec: Adding to iommu group 29
[ 8.670952] tegra-se 15840000.crypto: Adding to iommu group 27
[ 8.696919] scare-pigeon 13e00000.host1x:vi0-thi@15f00000: Adding to iommu group 30
[ 8.720743] scare-pigeon 13e00000.host1x:vi1-thi@14f00000: Adding to iommu group 31
[ 8.732442] scare-pigeon 13e00000.host1x:isp-thi@14b00000: Adding to iommu group 32
[ 8.925899] pva 16000000.pva0: Adding to iommu group 33
[ 8.929797] nvdla 15880000.nvdla0: Adding to iommu group 34
[ 8.946589] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx0: Adding to iommu group 35
[ 8.946939] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx0: initialized (streamid=18, iommu=smmu.0x0000000008000000)
[ 8.947142] nvdla 158c0000.nvdla1: Adding to iommu group 36
[ 8.951226] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx1: Adding to iommu group 37
[ 8.951925] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx1: initialized (streamid=19, iommu=smmu.0x0000000008000000)
[ 8.954898] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx2: Adding to iommu group 38
[ 8.955304] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx2: initialized (streamid=20, iommu=smmu.0x0000000008000000)
[ 8.961167] tegra-vic 15340000.vic: Adding to iommu group 39
[ 8.961275] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx3: Adding to iommu group 40
[ 8.961391] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx3: initialized (streamid=21, iommu=smmu.0x0000000008000000)
[ 8.967763] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx4: Adding to iommu group 41
[ 8.968855] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx4: initialized (streamid=22, iommu=smmu.0x0000000008000000)
[ 8.979680] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx5: Adding to iommu group 42
[ 8.980702] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx5: initialized (streamid=23, iommu=smmu.0x0000000008000000)
[ 8.983788] tegra-nvdec 15480000.nvdec: Adding to iommu group 43
[ 8.991617] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx6: Adding to iommu group 44
[ 8.991875] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx6: initialized (streamid=24, iommu=smmu.0x0000000008000000)
[ 8.998050] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx7: Adding to iommu group 46
[ 8.998151] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx7: initialized (streamid=25, iommu=smmu.0x0000000008000000)
[ 9.000414] tegra-nvenc 154c0000.nvenc: Adding to iommu group 45
[ 9.009467] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 47
[ 9.017381] tegra-nvjpg 15380000.nvjpg: Adding to iommu group 48
[ 9.026991] tegra-nvjpg 15540000.nvjpg: Adding to iommu group 49
[ 9.040924] tegra-ofa 15a50000.ofa: Adding to iommu group 50
[ 9.122036] tegra-dce d800000.dce: Adding to iommu group 51
[ 9.137187] tegra194-vi5 13e00000.host1x:vi0@15c00000: Adding to iommu group 30
[ 9.137237] tegra194-isp5 14800000.isp: Adding to iommu group 32
[ 9.138989] tegra194-vi5 13e00000.host1x:vi1@14c00000: Adding to iommu group 31
[ 9.148886] nv_platform 13800000.display: Adding to iommu group 52
[ 9.152402] platform 13800000.display:nvdisplay-niso: Adding to iommu group 53
[ 9.289942] spi-tegra114 3210000.spi: Adding to iommu group 1
[ 9.336152] spi-tegra114 3230000.spi: Adding to iommu group 1
[ 9.349021] tegra-hda 3510000.hda: Adding to iommu group 54
[ 9.455374] hailo1x 0005:01:00.0: Adding to iommu group 6
[ 9.456244] hailo1x 0005:01:00.0: Probing: Using specialized dma_ops=iommu_dma_ops
[ 9.579606] tegra234-aon c000000.aon: Adding to iommu group 55
[ 11.437702] nvethernet 6800000.ethernet: Adding to iommu group 56
[ 11.438056] nvethernet 6800000.ethernet: failed to read skip mac reset flag, default 0
[ 11.438061] nvethernet 6800000.ethernet: setting to default DMA bit mask
[ 11.438069] nvethernet 6800000.ethernet: failed to read UPHY GBE mode- default to 10G
[ 11.499177] rtl88x2ce 0001:01:00.0: Adding to iommu group 4
[ 11.557013] tegra210-admaif 290f000.admaif: Adding to iommu group 57
[ 13.547201] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

Best regards,
Junyong

Hi Michael,

I have an important update. I tested the same Hailo-10H with HailoRT 5.3.0 on a different board and it works perfectly.

Comparison:

AGX Orin (FAIL)

  • JetPack 6.2.2 / L4T R36.5.0
  • Kernel: 5.15.185-tegra
  • HailoRT: 5.3.0
  • scan: success
  • fw-control identify: FAIL (HAILO_COMMUNICATION_CLOSED, error 62)

Orin Nano Super (PASS)

  • JetPack 6.1.0 / L4T R36.4.4
  • Kernel: 5.15.148-tegra
  • HailoRT: 5.3.0
  • scan: success
  • fw-control identify: SUCCESS
    • Firmware Version: 5.3.0 (release,app)
    • Device Architecture: HAILO10H

Same Hailo-10H hardware, same HailoRT 5.3.0.
The only difference is JetPack/L4T version.

Is there a known compatibility issue between HailoRT 5.3.0 and JetPack 6.2.2 (L4T R36.5.0)?
Should I downgrade to JetPack 6.1.0 (L4T R36.4.4)?

Best regards,
Junyong

Hi @Junyong_Shin,

Great finding - we will investigate internally!
In the meanwhile since the same Hailo-10H works on JetPack 6.1.0, I’d recommend flashing your AGX Orin with JetPack 6.1.0 (L4T R36.4.4) as well, which is currently the best workaround for this L4T R36.5.0 compatibility issue.

Thanks,

Hi Michael,

Problem solved!

Downgrading from JetPack 6.2.2 (L4T R36.5.0) to
JetPack 6.1 GA (L4T R36.4.0) resolved the issue.

hailortcli fw-control identify now works perfectly:

  • Firmware Version: 5.3.0 (release,app)
  • Device Architecture: HAILO10H

This confirms a compatibility issue between
HailoRT 5.3.0 and JetPack 6.2.2 (L4T R36.5.0).

Thank you for your help!
Best regards,
Junyong

1 Like