Hailo-10H HAILO_COMMUNICATION_CLOSED(62) on Jetson AGX Orin with HailoRT 5.3.0

Hi,

I’m experiencing HAILO_COMMUNICATION_CLOSED(62) error with Hailo-10H on
Jetson AGX Orin. All software components are correctly installed and the
firmware loads successfully, but hailortcli fw-control identify fails.

Environment

  • Hardware: NVIDIA Jetson AGX Orin Developer Kit (64GB)
  • Hailo device: Hailo-10H (M.2, via PCIe expansion board)
  • OS: Ubuntu 22.04.5 LTS
  • Kernel: 5.15.185-tegra (aarch64)
  • JetPack: 6.2.2 (L4T R36.5.0)
  • CUDA: 12.6
  • HailoRT: 5.3.0
  • Driver: hailo1x_pci 5.3.0 (DKMS built)

What works

PCIe device recognized:
$ lspci | grep -i hailo
0005:01:00.0 Co-processor: Hailo Technologies Ltd. Device 45c4 (rev 01)

Firmware loads successfully (~1.7 seconds):
[10.346632] hailo1x 0005:01:00.0: SOC Firmware Batch loaded successfully
[10.346640] hailo1x 0005:01:00.0: Firmware loaded in 1724 ms
[10.363675] hailo1x 0005:01:00.0: Device created at /dev/h1x-0

Device files created:
/dev/h1x-0 (crw-rw-rw-)
/dev/hailo0 → /dev/h1x-0 (symlink)

hailortcli scan works:
$ hailortcli scan
Hailo Devices:
[-] Device: pci/0005:01:00.0

What fails

$ hailortcli fw-control identify
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62) - Failed to set default notification callbacks for Device HRPC-client
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_COMMUNICATION_CLOSED(62)

Firmware boot sequence (from dmesg)

The firmware loads the following files in sequence:

  1. customer_certificate.bin
  2. scu_fw.bin
  3. u-boot-0.dtb.signed
  4. u-boot-spl.bin
  5. fitImage (11MB Linux kernel image)
  6. image-fs (100MB Linux root filesystem)

This looks like a full Linux boot sequence rather than NPU-only mode.
Is this expected for Hailo-10H? Or should there be a different firmware
for NPU-only mode?

Attempted fixes (all failed)

  1. Module reload (modprobe -r / modprobe hailo1x_pci)
  2. force_hailo_pcie_boot_mode_over_bars=1
  3. force_hailo10h_legacy_mode=1
  4. Cold boot (DC adapter disconnected for 1+ minute)
  5. USB virtual ethernet activation (l4tbr0, usb0, usb1) - NO-CARRIER

Network interfaces (no Hailo Linux connection found)

usb0: <NO-CARRIER,BROADCAST,MULTICAST,UP> master l4tbr0 state DOWN
usb1: <NO-CARRIER,BROADCAST,MULTICAST,UP> master l4tbr0 state DOWN

ping 192.168.55.2 → 100% packet loss
ping 192.168.7.2 → 100% packet loss

Driver info

$ modinfo hailo1x_pci
filename: /lib/modules/5.15.185-tegra/updates/dkms/hailo1x_pci.ko
version: 5.3.0
vermagic: 5.15.185-tegra SMP preempt mod_unload modversions aarch64

Questions

  1. Is the firmware boot sequence (fitImage + image-fs) expected for
    Hailo-10H in NPU mode, or is this a Linux/AI Vision Module mode?
  2. If this is Linux mode, how do I switch to NPU mode for use with
    hailortcli and HailoRT API?
  3. Are there specific configurations required for Hailo-10H on
    Jetson AGX Orin with JetPack 6.2.2?

I found a similar case where downgrading HailoRT version resolved the issue (4.19→4.18 on Windows). Is there a known compatibility issue between HailoRT 5.3.0 and Hailo-10H firmware on Jetson platforms?

Any help would be greatly appreciated. Thank you.

Hi @Junyong_Shin,

  1. Is fitImage + image-fs expected? Yes - Hailo-10H is a SoC that boots an internal Linux running hailort_server.
  2. How to switch to NPU mode? You’re already in it - there is no separate NPU-only mode, the internal Linux manages the NPU via RPC over PCIe.
  3. Specific configs for Jetson AGX Orin + JetPack 6.2.2? No Jetson-specific docs exist, but load the driver with no_power_mode=Y to prevent D3 sleep (regarding “device visible but communication closed” please see here: Hailortcli fw-control identify: error 6).
  4. Version downgrade applicable? No - Hailo-10H requires HailoRT 5.x; the 4.x line is for Hailo-8/8L only and cannot be used.

Thanks,

(post deleted by author)

Hi Michael, Thank you for the suggestions. I tried all approaches but errors persist. Here are the detailed results:

**1. no_power_mode=Y**
```bash
sudo modprobe -r hailo1x_pci
sudo modprobe hailo1x_pci no_power_mode=Y
sleep 10
hailortcli fw-control identify
```
Result: HAILO_COMMUNICATION_CLOSED(62) — no change

**2. Force D0 power state**
```bash
echo on | sudo tee /sys/bus/pci/devices/0005:01:00.0/power/control
```
Result: HAILO_COMMUNICATION_CLOSED(62) — no change

**3. PCIe reset**
```bash
echo 1 | sudo tee /sys/bus/pci/devices/0005:01:00.0/reset
```
Result: Error changed to HAILO_DRIVER_OPERATION_FAILED(36)

Failed to open device file /dev/h1x-0 with error 6 Failed to connect to server


**4. Cold boot (full DC power disconnect, 1+ minute)**
After PCIe reset (error 36 state), disconnected DC adapter completely, 
waited 1 minute, reconnected and booted.
Result: Reverted back to HAILO_COMMUNICATION_CLOSED(62)

**Key Observations**
- Firmware always loads successfully:

SOC Firmware Batch loaded successfully Firmware loaded in 1724 ms Device created at /dev/h1x-0

- Error 36 "Failed to connect to server" only appears after PCIe reset, 
  before cold boot
- Cold boot always reverts to error 62
- hailort_server inside Hailo-10H appears to not start or be unreachable

Questions

1. Is there any way to verify if hailort_server is actually running inside Hailo-10H after firmware boot?

2. Are there any methods to extract logs from Hailo-10H internal Linux?

3. Could this be a hardware issue with this specific Hailo-10H unit?

Thank you.

Best regards, Junyong

Hi @Junyong_Shin,

  1. Verifying hailort_server inside Hailo-10H - The fact that hailortcli scan finds the device but fw-control identify returns HAILO_COMMUNICATION_CLOSED tells us the PCIe link is up but the RPC channel isn’t establishing - which could be the server not reaching ready state, or something blocking the shared-memory communication path between host and device.
  2. Extracting logs from Hailo-10H internal Linux - Not possible through any user-facing interface.
  3. Hardware defect - Unlikely given that PCIe enumeration, firmware loading, and device file creation all succeed. The link is healthy.

Can you please run:
sudo dmesg | grep -iE "smmu|iommu|fault|arm-smmu"
This might be useful to further diagnose the issue.

Thanks,

Hi Michael,

Here are the dmesg results:

Key Hailo-related entries:
[9.455374] hailo1x 0005:01:00.0: Adding to iommu group 6
[9.456244] hailo1x 0005:01:00.0: Probing: Using specialized dma_ops=iommu_dma_ops

No SMMU faults or IOMMU errors detected.
ARM-SMMU (SMMUv2) is present but no fault entries appear for the Hailo device.

Full output:
[ 0.000822] pid_max: default: 32768 minimum: 301
[ 0.180114] iommu: Default domain type: Translated
[ 0.180120] iommu: DMA domain TLB invalidation policy: strict mode
[ 0.234973] PCI: CLS 0 bytes, default 64
[ 0.289021] arm-smmu 8000000.iommu: probing hardware configuration…
[ 0.289025] arm-smmu 8000000.iommu: SMMUv2 with:
[ 0.289032] arm-smmu 8000000.iommu: stage 1 translation
[ 0.289035] arm-smmu 8000000.iommu: stage 2 translation
[ 0.289037] arm-smmu 8000000.iommu: nested translation
[ 0.289042] arm-smmu 8000000.iommu: stream matching with 128 register groups
[ 0.289046] arm-smmu 8000000.iommu: 128 context banks (0 stage-2 only)
[ 0.289059] arm-smmu 8000000.iommu: Supported page sizes: 0x61311000
[ 0.289063] arm-smmu 8000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.289065] arm-smmu 8000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.290050] arm-smmu 10000000.iommu: probing hardware configuration…
[ 0.290054] arm-smmu 10000000.iommu: SMMUv2 with:
[ 0.290056] arm-smmu 10000000.iommu: stage 1 translation
[ 0.290056] arm-smmu 10000000.iommu: stage 2 translation
[ 0.290057] arm-smmu 10000000.iommu: nested translation
[ 0.290060] arm-smmu 10000000.iommu: stream matching with 128 register groups
[ 0.290063] arm-smmu 10000000.iommu: 128 context banks (0 stage-2 only)
[ 0.290069] arm-smmu 10000000.iommu: Supported page sizes: 0x61311000
[ 0.290070] arm-smmu 10000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.290071] arm-smmu 10000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.290410] arm-smmu 12000000.iommu: probing hardware configuration…
[ 0.290414] arm-smmu 12000000.iommu: SMMUv2 with:
[ 0.290417] arm-smmu 12000000.iommu: stage 1 translation
[ 0.290417] arm-smmu 12000000.iommu: stage 2 translation
[ 0.290418] arm-smmu 12000000.iommu: nested translation
[ 0.290421] arm-smmu 12000000.iommu: stream matching with 128 register groups
[ 0.290424] arm-smmu 12000000.iommu: 128 context banks (0 stage-2 only)
[ 0.290429] arm-smmu 12000000.iommu: Supported page sizes: 0x61311000
[ 0.290430] arm-smmu 12000000.iommu: Stage-1: 48-bit VA → 48-bit IPA
[ 0.290432] arm-smmu 12000000.iommu: Stage-2: 48-bit IPA → 48-bit PA
[ 0.323532] tegra-bpmp bpmp: Adding to iommu group 0
[ 2.382951] tegra-gpcdma 2600000.dma-controller: Adding to iommu group 1
[ 2.391065] tegra-i2c 3160000.i2c: Adding to iommu group 1
[ 2.395293] tegra-i2c 3180000.i2c: Adding to iommu group 1
[ 2.396776] tegra-i2c 3190000.i2c: Adding to iommu group 1
[ 2.398127] tegra-i2c 31b0000.i2c: Adding to iommu group 1
[ 2.399435] tegra-i2c 31c0000.i2c: Adding to iommu group 1
[ 2.400749] tegra-i2c 31e0000.i2c: Adding to iommu group 1
[ 2.402359] tegra-i2c c240000.i2c: Adding to iommu group 1
[ 2.404128] tegra-i2c c250000.i2c: Adding to iommu group 1
[ 2.432350] sdhci-tegra 3460000.mmc: Adding to iommu group 2
[ 2.433408] sdhci-tegra 3400000.mmc: Adding to iommu group 3
[ 5.768299] tegra194-pcie 14100000.pcie: Adding to iommu group 4
[ 5.904213] pcieport 0001:00:00.0: Adding to iommu group 4
[ 5.905416] tegra194-pcie 14160000.pcie: Adding to iommu group 5
[ 6.043978] pcieport 0004:00:00.0: Adding to iommu group 5
[ 6.044317] nvme 0004:01:00.0: Adding to iommu group 5
[ 6.045070] tegra194-pcie 141a0000.pcie: Adding to iommu group 6
[ 6.050561] nvme nvme0: 12/0/0 default/read/poll queues
[ 6.280467] pcieport 0005:00:00.0: Adding to iommu group 6
[ 6.282452] tegra-xusb 3610000.usb: Adding to iommu group 7
[ 6.298718] tegra-xudc 3550000.usb: Adding to iommu group 8
[ 7.382965] systemd[1]: systemd 249.11-0ubuntu3.20 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 7.803968] systemd[1]: Queued start job for default target Graphical Interface.
[ 8.140481] tegra-host1x 13e00000.host1x: Adding to iommu group 9
[ 8.141621] host1x-context host1x-ctx.0: Adding to iommu group 10
[ 8.141761] host1x-context host1x-ctx.1: Adding to iommu group 11
[ 8.141876] host1x-context host1x-ctx.2: Adding to iommu group 12
[ 8.141992] host1x-context host1x-ctx.3: Adding to iommu group 13
[ 8.142118] host1x-context host1x-ctx.4: Adding to iommu group 14
[ 8.142231] host1x-context host1x-ctx.5: Adding to iommu group 15
[ 8.142347] host1x-context host1x-ctx.6: Adding to iommu group 16
[ 8.142474] host1x-context host1x-ctx.7: Adding to iommu group 17
[ 8.142593] host1x-context host1x-ctx.8: Adding to iommu group 18
[ 8.142703] host1x-context host1x-ctx.9: Adding to iommu group 19
[ 8.142817] host1x-context host1x-ctx.10: Adding to iommu group 20
[ 8.142927] host1x-context host1x-ctx.11: Adding to iommu group 21
[ 8.143027] host1x-context host1x-ctx.12: Adding to iommu group 22
[ 8.143139] host1x-context host1x-ctx.13: Adding to iommu group 23
[ 8.143246] host1x-context host1x-ctx.14: Adding to iommu group 24
[ 8.143359] host1x-context host1x-ctx.15: Adding to iommu group 25
[ 8.659107] tegra-se 15820000.crypto: Adding to iommu group 26
[ 8.670180] tegra-soc-hwpm f100000.hwpm: Adding to iommu group 28
[ 8.670226] tsec 15500000.tsec: Adding to iommu group 29
[ 8.670952] tegra-se 15840000.crypto: Adding to iommu group 27
[ 8.696919] scare-pigeon 13e00000.host1x:vi0-thi@15f00000: Adding to iommu group 30
[ 8.720743] scare-pigeon 13e00000.host1x:vi1-thi@14f00000: Adding to iommu group 31
[ 8.732442] scare-pigeon 13e00000.host1x:isp-thi@14b00000: Adding to iommu group 32
[ 8.925899] pva 16000000.pva0: Adding to iommu group 33
[ 8.929797] nvdla 15880000.nvdla0: Adding to iommu group 34
[ 8.946589] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx0: Adding to iommu group 35
[ 8.946939] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx0: initialized (streamid=18, iommu=smmu.0x0000000008000000)
[ 8.947142] nvdla 158c0000.nvdla1: Adding to iommu group 36
[ 8.951226] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx1: Adding to iommu group 37
[ 8.951925] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx1: initialized (streamid=19, iommu=smmu.0x0000000008000000)
[ 8.954898] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx2: Adding to iommu group 38
[ 8.955304] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx2: initialized (streamid=20, iommu=smmu.0x0000000008000000)
[ 8.961167] tegra-vic 15340000.vic: Adding to iommu group 39
[ 8.961275] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx3: Adding to iommu group 40
[ 8.961391] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx3: initialized (streamid=21, iommu=smmu.0x0000000008000000)
[ 8.967763] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx4: Adding to iommu group 41
[ 8.968855] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx4: initialized (streamid=22, iommu=smmu.0x0000000008000000)
[ 8.979680] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx5: Adding to iommu group 42
[ 8.980702] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx5: initialized (streamid=23, iommu=smmu.0x0000000008000000)
[ 8.983788] tegra-nvdec 15480000.nvdec: Adding to iommu group 43
[ 8.991617] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx6: Adding to iommu group 44
[ 8.991875] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx6: initialized (streamid=24, iommu=smmu.0x0000000008000000)
[ 8.998050] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx7: Adding to iommu group 46
[ 8.998151] pva_iommu_context_dev 16000000.pva0:pva0_niso1_ctx7: initialized (streamid=25, iommu=smmu.0x0000000008000000)
[ 9.000414] tegra-nvenc 154c0000.nvenc: Adding to iommu group 45
[ 9.009467] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 47
[ 9.017381] tegra-nvjpg 15380000.nvjpg: Adding to iommu group 48
[ 9.026991] tegra-nvjpg 15540000.nvjpg: Adding to iommu group 49
[ 9.040924] tegra-ofa 15a50000.ofa: Adding to iommu group 50
[ 9.122036] tegra-dce d800000.dce: Adding to iommu group 51
[ 9.137187] tegra194-vi5 13e00000.host1x:vi0@15c00000: Adding to iommu group 30
[ 9.137237] tegra194-isp5 14800000.isp: Adding to iommu group 32
[ 9.138989] tegra194-vi5 13e00000.host1x:vi1@14c00000: Adding to iommu group 31
[ 9.148886] nv_platform 13800000.display: Adding to iommu group 52
[ 9.152402] platform 13800000.display:nvdisplay-niso: Adding to iommu group 53
[ 9.289942] spi-tegra114 3210000.spi: Adding to iommu group 1
[ 9.336152] spi-tegra114 3230000.spi: Adding to iommu group 1
[ 9.349021] tegra-hda 3510000.hda: Adding to iommu group 54
[ 9.455374] hailo1x 0005:01:00.0: Adding to iommu group 6
[ 9.456244] hailo1x 0005:01:00.0: Probing: Using specialized dma_ops=iommu_dma_ops
[ 9.579606] tegra234-aon c000000.aon: Adding to iommu group 55
[ 11.437702] nvethernet 6800000.ethernet: Adding to iommu group 56
[ 11.438056] nvethernet 6800000.ethernet: failed to read skip mac reset flag, default 0
[ 11.438061] nvethernet 6800000.ethernet: setting to default DMA bit mask
[ 11.438069] nvethernet 6800000.ethernet: failed to read UPHY GBE mode- default to 10G
[ 11.499177] rtl88x2ce 0001:01:00.0: Adding to iommu group 4
[ 11.557013] tegra210-admaif 290f000.admaif: Adding to iommu group 57
[ 13.547201] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

Best regards,
Junyong