Hailo-10H on Raspberry Pi 5: how is the on-chip `pci_ep` driver updated?

stuck at 5.1.1 after upgrading host stack to 5.3.0

Hardware: Raspberry Pi 5 (16 GB), Raspberry Pi AI HAT+ 2 (Hailo-10H), board SKU-ID 6
OS: NixOS, kernel linux-rpi 6.18.39, aarch64


I have a working 5.1.1 stack and wanted to move to 5.3.0 for the newer model
zoo. Every host-side component upgraded cleanly, but the chip continues to
report a 5.1.1 pci_ep driver and the version check then fails every ioctl.

What I upgraded, all to 5.3.0:

  • hailort-drivers, tag v5.3.0-hotfix-kernel-above-6.15, built from source
    against my running kernel. Confirmed the built module is 5.3.0: dmesg
    reports hailo1x: Init module. driver version 5.3.0, and
    HAILO_DRV_VER_{MAJOR,MINOR,REVISION} in common/hailo_ioctl_common.h read
    5, 3, 0.
  • Firmware from
    https://hailo-hailort.s3.eu-west-2.amazonaws.com/Hailo10H/5.3.0/FW/hailo10h_fw.tar.gz.
    Confirmed this is genuinely different content from 5.1.1’s tarball, not a
    republish: image-fs and fitImage differ by sha256 between the two
    versions. I install all eighteen files from the archive, including the
    u-boot-{9,10,11,12,13,14}.dtb.signed and u-boot-default.dtb.signed that
    5.1.1 did not ship.
  • hailort 5.3.0 (hailort_5.3.0_arm64.deb from dev-public.hailo.ai).
  • hailo_gen_ai_model_zoo 5.3.0.

What happens at boot:

The firmware loads through all three stages with no errors:

hailo1x 0001:01:00.0: File hailo/hailo10h/customer_certificate.bin written successfully
hailo1x 0001:01:00.0: File hailo/hailo10h/scu_fw.bin written successfully
hailo1x 0001:01:00.0: Board SKU-ID is: 6
hailo1x 0001:01:00.0: File hailo/hailo10h/u-boot-6.dtb.signed written successfully
hailo1x 0001:01:00.0: Firmware batch programming completed for stage 3
hailo1x 0001:01:00.0: vDMA transfer completed, triggering boot
hailo1x 0001:01:00.0: SOC Firmware Batch loaded successfully
hailo1x 0001:01:00.0: Firmware loaded in 2261 ms
hailo1x 0001:01:00.0: Mismatch Driver version pcie driver 5:3:0 pci_ep driver 5:1:1
hailo1x 0001:01:00.0: hailo_soc_get_driver_info has failed with err -22
hailo1x 0001:01:00.0: Device created at /dev/h1x-0
hailo1x 0001:01:00.0: driver_compatible is false

Note this is the mismatch path in hailo_soc_get_driver_info(), not the
error path: soc_control() returned successfully and the chip answered
HAILO_PCIE_SOC_CONTROL_CODE_QUERY_DRIVER_INFO with a real 5.1.1. It is not
a failed query returning garbage.

Consequence: every ioctl fails with EINVAL.

$ hailortcli fw-control identify
[HailoRT] [error] Ioctl HAILO_SOC_CONNECT failed with 22. Read dmesg log for more info
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed soc_connect

hailo-ollama starts and serves, but every /api/generate returns 500
“LLM not loaded”. hailortcli logs system_control also fails, since that path
goes through soc_connect too, so I cannot read the chip’s own log to see
what it booted.

What I have ruled out:

  • Stale firmware content (the two tarballs differ by hash)
  • A partial firmware install (all eighteen files present in the search path)
  • A stale build of the driver (version constants and dmesg both say 5.3.0)
  • A confused chip (full power cycle, not just a reboot, between every attempt)

My questions:

  1. Is the pci_ep driver resident in the Hailo-10H’s own flash, rather than
    being supplied by the blobs the PCIe driver pushes at probe? The behaviour
    above suggests the pushed firmware is not what ends up running, or that
    pci_ep comes from somewhere the tarball does not touch.

  2. If so, what updates it? hailortcli on the H10 has no fw-update
    subcommand (only identify under fw-control), so whatever performs this
    is not in the public hailort package. Is there a separate tool, and is it
    available outside the Developer Zone?

  3. Is a 5.1.1 chip with a 5.3.0 host stack simply unsupported, and the
    intended upgrade path something other than replacing the files in
    /lib/firmware/hailo/hailo10h/?

  4. Is this specific to the Raspberry Pi AI HAT+ 2, where the board may ship
    with a factory-flashed endpoint driver, as opposed to an M.2 card or SoM?

Happy to provide full dmesg, the exact file list installed, or sha256s of
any firmware file. I am running this from a declarative NixOS configuration,
so every version and artifact hash is pinned and reproducible on request.

After I posted this, I went on to try other permutations. I had a brutal session with a frontier model, trying to figure out how to patch this to work and here are the battle scars that I came away with so you don’t have to (if you are also looking to build a deterministic nixos pi with a Kailo10H APU:

# docs/hailo.md

The Hailo-10H AI HAT+ 2 on a Raspberry Pi 5, under NixOS. Everything learned

bringing it up on 2026-09-13, including what did not work and why.

`[V]` verified on real hardware · `[S]` read from source · `[?]` unresolved

—

## 1. Which HAT you have decides everything

Two products share a name and only one runs LLMs.

| | AI HAT+ | AI HAT+ 2 |

|—|—|—|

| Chip | Hailo-8L / Hailo-8 | **Hailo-10H** |

| TOPS | 13 / 26 | 40 (INT4) |

| Own memory | none, uses the Pi’s | **8 GB onboard** |

| LLM support | **Not supported** (Raspberry Pi’s own docs) | yes, up to ~6B params |

| Driver | `hailort-pcie-driver`, `hailo_pci` | `h10-hailort-pcie-driver`, `hailo1x_pci` |

The two driver stacks **cannot coexist**; the Debian metapackages explicitly

conflict. Everything in this document is the Hailo-10H.

If you have the original AI HAT+, it is a vision accelerator. It will not run

Llama no matter what you install, and the LLM marketing you may have read

applies to the other board.

—

## 2. The M.2 slot is disabled in the stock device tree

`[V]` **The single most time-wasting gotcha in the whole bring-up.**

The Pi 5 has three PCIe controllers. `pcie@1000120000` (label `pciex4`) carries

the RP1 south bridge for USB and ethernet and comes up on its own. The M.2

connector is `pcie@1000110000` (label `pciex1`) and its `status` is

`“disabled”` until something says otherwise.

Symptom: `lspci` shows exactly two devices, both the Pi’s own, and no amount of

driver work changes it. The HAT is electrically connected and completely

invisible.

On Raspberry Pi OS this is `dtparam=pciex1` in `config.txt`. NixOS has no

`config.txt`. It is a device tree overlay, in `hosts/.nix`:

```nix

hardware.deviceTree.overlays = [

{

name = “pcie-external”;

dtsText = ‘’

    /dts-v1/;

    /plugin/;



    / {

      compatible = "brcm,bcm2712";



      fragment@0 {

        target = <&pciex1>;

        \__overlay_\_ {

          status = "okay";

          max-link-speed = <3>;

        };

      };

    };

  '';

}

];

```

`max-link-speed = <3>` is `dtparam=pciex1_gen=3`. Raspberry Pi does not certify

Gen 3 on this connector; it negotiated cleanly here (`link up, 8.0 GT/s PCIe x1`)

and if it proves unstable, drop the property for Gen 2.

`nixos-hardware`'s `raspberry-pi-5` profile does **not** enable this. Nothing in

it mentions PCIe beyond an initrd module name.

**Device tree changes need a real boot.** `./build test` is not enough; the

bootloader config has to be rewritten, which means `boot` or `switch` followed by

a reboot.

**Verify the label before writing the overlay** rather than trusting this

document. Decompile the DTB the row actually builds:

```fish

set dtb (nix build --no-link --print-out-paths .#nixosConfigurations..config.hardware.deviceTree.package)/broadcom/bcm2712-rpi-5-b.dtb

nix shell nixpkgs#dtc --command dtc -I dtb -O dts $dtb | grep -n “pciex\|pcie@”

```

Success looks like:

```

0001:00:00.0 PCI bridge [0604]: Broadcom BCM2712 PCIe Bridge [14e4:2712]

0001:01:00.0 Co-processor [0b40]: Hailo Technologies Ltd. Hailo-10H AI Processor [1e60:45c4]

```

—

## 3. The stack, and the five things that must agree

```

device tree overlay enables pcie@1000110000

→ hailo1x_pci.ko out-of-tree, built against this row’s kernel

→ firmware blobs 3 stages, ~2.2s, SKU read from the chip

→ /dev/hailo0 (5.1.1) or /dev/h1x-0 (5.3.0+)

→ libhailort userspace runtime

→ hailo-ollama :8000 ollama-compatible API, from the GenAI model zoo

```

Four components ship from Hailo and must match:

| component | source | pinned by |

|—|—|—|

| driver | `github.com/hailo-ai/hailort-drivers`, tagged | the zoo |

| firmware | `hailo-hailort.s3.eu-west-2.amazonaws.com` | independent, see §6 |

| runtime | a `.deb` | the zoo |

| model zoo | a `.deb` | **the constraint** |

**The zoo is the constraint** because it is the only thing that provides

`hailo-ollama`, and it exists only at the versions Hailo publishes it at.

And a **fifth**, which is the unresolved part: the `pci_ep` driver running *on

the chip*. See §6.

—

## 4. Where the packages actually live

`[V]` None of this is guessable; all of it was found by probing.

**Driver**: GitHub, tagged. `git ls-remote --tags` lists them. Note

`v5.3.0-hotfix-kernel-above-6.15` exists separately from `v5.3.0`, for kernels

above 6.15.

**Firmware**: `https://hailo-hailort.s3.eu-west-2.amazonaws.com/Hailo10H//FW/hailo10h_fw.tar.gz`

Public, no login. Version interpolates cleanly. 5.1.1, 5.3.0 and 5.4.0 all exist.

**Runtime**, and the name changed at 5.x:

- 5.1.1: `h10-hailort_5.1.1_arm64.deb` from `archive.raspberrypi.com/debian/pool/main/h/h10-hailort/`

- 5.3.0: `hailort_5.3.0_arm64.deb` from `dev-public.hailo.ai/2026_04/Hailo10/`

At 5.x Hailo merged the Hailo-8 and Hailo-10 runtimes into one `hailort`

package. The deb’s control fields say `Conflicts/Breaks/Replaces: hailort`,

confirming a merge rather than a parallel package. Hailo’s S3 bucket returns

**403** for the 5.1.1 runtime; the Raspberry Pi archive is the only public

source for it.

**Model zoo**: `https://dev-public.hailo.ai//Hailo10/hailo_gen_ai_model_zoo__arm64.deb`

**The date segment is not derivable from the version.** Discovered by probing:

| version | date segment |

|—|—|

| 5.1.1 | `2025_12` |

| 5.2.0 | `2026_01` |

| 5.3.0 | `2026_04` |

| 5.4.0 | **not published** (firmware is, runtime and zoo are not) |

To find a new one:

```fish

for d in 2026_05 2026_06 2026_07 2026_08

for v in 5.4.0 5.5.0

echo -n "$d/$v: "

curl -sI "https://dev-public.hailo.ai/$d/Hailo10/hailo_gen_ai_model_zoo_$v"\_arm64.deb | head -1

end

end

```

The S3 bucket returns 403 rather than 404 because it denies listing. A wrong

path and a private one look identical there. `dev-public.hailo.ai` returns

honest 404s.

—

## 5. Model names change between zoo versions

`[V]` A name from the wrong version returns **404 from `/api/pull`** while

`GET /hailo/v1/list` may still advertise it, if the seeded manifests are stale.

That combination looks like an intermittent network problem and is not.

| 5.1.1 | 5.3.0 |

|—|—|

| `deepseek_r1_distill_qwen:1.5b` | `deepseek_r1:1.5b` |

| `llama3.2:3b` | `llama3.2:1b` |

| `qwen2:1.5b` | `qwen2:1.5b` |

| `qwen2.5-coder:1.5b` | `qwen2.5-coder:1.5b` |

| `qwen2.5-instruct:1.5b` | `qwen2.5:1.5b` |

| (none) | `qwen3:1.7b` |

Note 5.3.0 **drops llama3.2 at 3b**, the largest model 5.1.1 offers, and adds

`qwen3`. Whether that is an upgrade depends on what you want.

Always check `GET /hailo/v1/list` on the target after a version change.

—

## 6. `pci_ep`: the unresolved blocker `[?]`

**This is why the stack is pinned to 5.1.1.**

The chip runs its own endpoint driver. `hailo_soc_get_driver_info()` in

`linux/pcie/src/soc.c` queries it over PCIe and compares against the host

driver’s `HAILO_DRV_VER_*` constants. A mismatch sets

`board->soc.driver_compatible = false` and returns `-EINVAL`.

```

hailo1x: Mismatch Driver version pcie driver 5:3:0 pci_ep driver 5:1:1

hailo1x: hailo_soc_get_driver_info has failed with err -22

hailo1x: Device created at /dev/h1x-0

hailo1x: driver_compatible is false

```

Every ioctl then fails with EINVAL. `hailortcli` reports `Failed soc_connect`,

`hailo-ollama` starts and serves but returns 500 “LLM not loaded” for every

request, and **you cannot read the chip’s own log** because

`hailortcli logs system_control` also goes through `soc_connect`.

Note this is the *mismatch* path, not the *error* path in that function: the

query succeeded and the chip genuinely answered 5.1.1. It is not garbage from a

failed call.

### What was tested `[V]`

| firmware | host driver | result |

|—|—|—|

| 5.1.1 | 5.1.1 | works |

| 5.3.0 | 5.3.0 | mismatch, `pci_ep 5:1:1` |

| 5.4.0 | 5.1.1 | works |

| 5.4.0 | 5.3.0 | mismatch, `pci_ep 5:1:1` |

**Three different firmware versions all leave `pci_ep` reporting 5.1.1.**

Ruled out:

- Stale firmware content: 5.3.0’s and 5.4.0’s `image-fs` and `fitImage` differ

by sha256 from 5.1.1’s

- Partial install: all eighteen files from the 5.3.0 tarball present in the

firmware search path

- Stale driver build: `HAILO_DRV_VER_*` in `common/hailo_ioctl_common.h` read

5.3.0, and `dmesg` confirms `Init module. driver version 5.3.0`

- Confused chip: full power cycle, not a reboot, between every attempt

**Conclusion:** the blobs the PCIe driver pushes at probe do not supply

`pci_ep`. It is most likely flash-resident, or the chip boots from flash and

ignores what is pushed when flash is valid.

`hailortcli` on the H10 has only `fw-control identify`, no `fw-update`, so

whatever updates the endpoint driver is not in any public Hailo package.

**Open question for Hailo:** what updates `pci_ep`, and is that tool available

outside the Developer Zone? The table above is the whole argument and is hard

to misread; post it to `community.hailo.ai`.

Until answered, **the stack stays at 5.1.1** and 5.3.0 lives on a branch.

—

## 7. Building the driver

`[V]` Compiles clean against kernel 6.18.39 with no patches, despite 5.1.1’s

`LINUX_VERSION_CODE` guards topping out around 6.5.

Three traps:

**The variable is `KERNEL_DIR`, with the underscore.** `KERNELDIR` is ignored

silently and the build then fails on `/lib/modules/$(uname -r)/build`, which

reads like a missing kernel rather than a wrong variable name.

**Do not set `sourceRoot = “linux/pcie”`.** The Makefile there reaches up into

`../../common`, `../vdma` and `../utils` for most of its sources. With the build

directory set to the subdirectory those paths land outside it and the compiler

fails with `Permission denied` writing dependency files. Build from the tree

root and `make -C linux/pcie`.

**`make` with no target prints help.** The target is `all`.

**The udev rule moved.** 5.1.1 ships `linux/pcie/51-hailo-udev.rules` containing

`SUBSYSTEM==“hailo_chardev”, MODE=“0666”`. 5.3.0 ships no rules file and the

class changed: `pcie.c` calls `class_create_compat(“hailo1x”)`. Without a

matching rule the node is root-only and any non-root caller gets EACCES.

**The module is `hailo1x_pci`**, not `hailo_pci`.

**The device node renamed**: `/dev/hailo0` at 5.1.1, `/dev/h1x-0` at 5.3.0, from

`HAILO_DEV_NAME “h1x”` and a `“%s-%d”` format in `hailo_ioctl_common.h`.

—

## 8. The firmware

Three stages, observed:

```

stage 1 customer_certificate.bin

      scu_fw.bin              -> chip boots its SCU

stage 2 u-boot-.dtb.signed → SKU read FROM THE CHIP at probe

stage 3 u-boot-spl.bin, u-boot-tfa.itb, fitImage, image-fs

      batch-programmed over vDMA, then "triggering boot"

```

**The SKU is not known ahead of time.** The driver reads it from the board

(`Board SKU-ID is: 6` here) and asks for the matching DTB.

**`u-boot-tfa.itb` is requested but is not a file in the tarball.** Presumably

inside `fitImage`; the load succeeds anyway.

**Install every file. Never hardcode names.** 5.1.1 ships

`u-boot-{0,1,3,4,5,6}.dtb.signed` (note: no 2); 5.3.0 adds

`{9,10,11,12,13,14}` and `u-boot-default`. A list written against one version

silently installs a subset of the next and still builds, because every name it

does list still exists. A board whose SKU is one of the dropped numbers then

fails at stage 2 on a DTB that shipped and never reached the store.

```nix

installPhase = ‘’

mkdir -p $out/lib/firmware/hailo/hailo10h

install -m444 -t $out/lib/firmware/hailo/hailo10h \*

‘’;

```

—

## 9. The runtime and the zoo

Both are prebuilt arm64 Debian binaries. `dpkg -x` plus `autoPatchelfHook`.

**Libraries needed**, found by running the patched binary and reading each

failure in turn:

| | 5.1.1 | 5.3.0 |

|—|—|—|

| `libstdc++.so.6` | yes | yes |

| `libssl.so.3` | yes | yes |

| `libusb-1.0.so.0` | no | **yes** |

`libusb` at 5.3.0 is wanted by `libhailort` itself, not by `hailortcli`, so when

patching by hand the error appears on the binary that loads it and patching only

that binary does not help. A dependency’s rpath does not inherit from its loader.

**Drop the GStreamer plugin.** `lib/aarch64-linux-gnu/libgsthailo.so` is a

video-pipeline element wanting the whole GStreamer and GLib stack. An LLM row

never loads it, and `autoPatchelfHook` will otherwise fail the build on missing

GStreamer. `rm -rf $out/lib/aarch64-linux-gnu` in the install phase.

### How `hailo-ollama` finds things `[V]` via strace

Two separate XDG searches, and they do not behave the same way:

- **Config**: `$XDG_CONFIG_HOME`, then each `$XDG_CONFIG_DIRS` entry, each with

`/hailo-ollama` appended.

- **Data**: each `$XDG_DATA_DIRS` entry with **`/hailo-ollama`** appended for the

writable copy. The `/share/hailo-ollama` suffix appears in the strace but

applies to the store fallback.

That distinction cost hours. Seeding into `share/hailo-ollama` produced a server

that ignored the writable tree entirely, fell through to the store, and aborted

with “Read-only file system” on `models/blob`.

**The weights go somewhere else again**: `$HOME/.local/share/hailo-ollama/models/blob`.

**5.1.1 requires a config file**, `etc/xdg/hailo-ollama/hailo-ollama.json`, and

aborts with `hailo-ollama directory not found` without it. **5.3.0 ships no

config at all** and reads `OLLAMA_HOST` instead, defaulting to `0.0.0.0:8000`,

logging which it used:

```

MyApp:Using ‘OLLAMA_HOST’ server connection: 0.0.0.0:8000

```

**`makeWrapper --set-default`, never `–set`.** A plain `–set` wins over the

systemd unit’s `Environment=`, so the service cannot redirect `XDG_DATA_DIRS` at

a writable copy and fails on the read-only store. That is a full evening on its

own.

—

## 10. The service module

Traps in rough order of how long each took.

**`ExecStartPre` runs inside the namespace.** `ProtectSystem = “strict”` and any

`BindPaths` are set up *before* `ExecStartPre`, so a pre-hook cannot create a

directory that a `BindPaths` entry needs. The seeding has to be a **separate

unit**, running as root with no namespacing, `before` and `requiredBy` the

server.

**`ProtectSystem = “strict”` blocks binding over store paths.** A `BindPaths`

entry targeting `/nix/store/…` fails with “No such file or directory” even

though the path exists. Use `WorkingDirectory` and environment redirection

instead where possible.

**`DynamicUser` is wrong for a service owning persistent state.** It allocates a

transient UID per start, so anything a seed script chowns is owned by the wrong

UID by the time the service runs. Symptom: a tree owned by `nobody:nogroup` the

service cannot write into. Use a fixed system user. That also moves the state

directory from `/var/lib/private/` to `/var/lib/`.

**The seed guard must be version-aware.** Guarding on “does the directory exist”

seeds once and never again, so a zoo bump leaves the old manifests under a new

server: `/hailo/v1/list` advertises models `/api/pull` returns 404 for. Stamp the

zoo’s store path into the directory and re-seed on mismatch.

**The seed must also wipe `.local`.** The server writes pulled weights there and

the seed does not otherwise manage it. A version change left the old version’s

weights behind after everything else had moved, and the server failed on them

with the manifests looking correct.

**Recursive `chown` belongs inside the version guard.** Run unconditionally over

the whole state directory it walks every model blob, which on an SD card with a

few gigabytes takes minutes and makes every activation look frozen.

**`ConditionPathExists` fails silently.** A wrong device node name leaves the

unit `inactive (dead)` with `start condition unmet`, not failed. It will not

appear in `systemctl --failed`. Given the node renamed between 5.1.1 and 5.3.0,

this is a real hazard.

**The udev rule sets `MODE=“0666”`, no group.** So `SupplementaryGroups` is

unnecessary and setting it to a group that does not exist fails the unit with

`216/GROUP`. Worth being clear-eyed: 0666 means any local user can open the NPU.

That is Hailo’s choice, not one made here, and on a single-user headless box it

is not much of an exposure.

—

## 11. A power cycle is not a reboot

`[V]` **The chip holds firmware state across a warm reset.**

If a firmware load fails partway, the SCU is left running and waiting.

`rmmod`/`insmod` or a reboot then times out at stage 1 with `-110` (ETIMEDOUT)

and `boot_status ffffffff`, which looks like a different and worse bug. It is

not.

Pull power for ten seconds. This applies to any driver or firmware change.

Related: **`modprobe` after a `switch` loads from the booted generation’s depmod

index**, not the current one, so a newly deployed driver will not load with

`modprobe` and reports the old version. `insmod` on the explicit

`/run/current-system/…` path works, but a power cycle is cleaner.

—

## 12. The stale-hash trap

`[V]` Cost an evening, twice.

A fixed-output derivation is keyed on the **hash**, not the URL or the rev.

Bumping `version` and `rev` while leaving the old hash in place produces a

derivation *named* `hailo-pcie-driver-5.3.0` that contains 5.1.1’s source. It

builds cleanly, installs, loads, and logs `driver version 5.1.1` at boot.

Nothing errors anywhere.

Nix cannot catch this. It cannot know what is inside a fixed-output derivation

before building it, and once built the hash *is* the identity. That is the

property that makes offline builds work.

An eval assertion can catch the `version` strings disagreeing. Only a runtime

check catches the hash being wrong:

```fish

ssh ‘sudo dmesg | grep -iE “Init module|Mismatch|driver_compatible”’

```

after a **power cycle**.

When bumping anything:

```fish

nix-prefetch-url --unpack https://github.com/hailo-ai/hailort-drivers/archive/refs/tags/.tar.gz

nix hash convert --hash-algo sha256 --to sri

```

—

## 13. Version assertion: firmware is excluded deliberately

Driver, runtime and zoo share an ABI and must agree: the zoo’s `hailo-ollama`

links against the runtime’s `libhailort`, and the runtime’s ioctls must match the

driver’s.

Firmware is versioned on its own schedule, 5.4.0 firmware exists while the

5.4.0 runtime and zoo do not, and determines `pci_ep`, which must **satisfy**

the host driver rather than equal it.

So the assertion compares three, not four, and says why.

—

## 14. Performance, measured `[V]`

Raspberry Pi 5, 16 GB, AI HAT+ 2, same prompt (“Write two sentences about rain”).

| backend | model | tok/s |

|—|—|—|

| NPU | `qwen2.5-instruct:1.5b` | **6.1–6.5** |

| NPU | `llama3.2:3b` | **2.5** |

| CPU (ollama) | `gemma2:2b` | ~3 |

| CPU (ollama) | `llama3.1:8b` | **1.7** |

**The published numbers are optimistic.** Raspberry Pi’s marketing says 30–50

tok/s on Llama 3.2 1B; a hands-on project reported 7–10 on Qwen2-1.5B. The

latter is closer to reality.

**The first generate after a pull or a power cycle is much slower**, 15 seconds

where the second takes 6, because it loads the model onto the chip. Measure the

second run.

6.5 tok/s is slower than reading speed. It is fine for short structured work:

classification, extraction, a commit-message draft, a `home.target` hook. It is

not a chat assistant you will enjoy.

**Running both backends is the right answer.** They answer on different ports

(11434 CPU, 8000 NPU) and make opposite trades: three and a half times the speed

for a fifth of the parameters. `roles.llm.backends = [ “cpu” “hailo” ]`.

—

## 15. Compiling your own models `[?]`

The zoo offers five or six HEFs Hailo chose. Your own model is possible in

principle.

**[hailo-10h-llm-compiler]( GitHub - l-nmch/hailo-10h-llm-compiler: Compile your own LLM for the Hailo-10H — an open pipeline from Hugging Face checkpoint to a self-contained HEF, served with hailo-ollama. · GitHub )**

(MIT) compiles a Hugging Face checkpoint into a self-contained HEF that

`hailo-ollama` can serve. Six steps: ONNX export, HAR parse, graph surgery,

INT4/INT8 quantization, HEF compile, registration.

**Read its warnings before investing.** The author’s own words: "Highly

experimental, do not expect it to work." Specifically:

- **Multi-token generation is degraded.** Open issue: cache reads during

token-by-token inference return truncated tensors, ~30% of columns structurally

zeroed. Prefill is exact.

- **Validated on a 25M-parameter toy** (TinyStories-LLaMA2-25M). "Small models

only, so far."

- **Pins DFC 5.3.0**, which lands back in the `pci_ep` problem.

- **Docker, ~30 GB, GPU recommended** for the quantization step.

It also needs Hailo’s **Dataflow Compiler**, which is proprietary, x86_64 Linux

only, and behind a free Developer Zone registration. The pipeline around it is

MIT; the compiler is not.

Not a path to better models today. A genuinely interesting project to contribute

to.

—

## 16. Licensing

| component | licence |

|—|—|

| PCIe driver | GPL-2.0 |

| HailoRT runtime | MIT |

| Firmware blobs | proprietary, no source, terms unstated |

| GenAI model zoo | proprietary |

| Dataflow Compiler | proprietary, login-walled |

Two of these need permitting in nixpkgs. Name them rather than using

`allowUnfree = true`:

```nix

nixpkgs.config.allowUnfreePredicate =

pkg:

builtins.elem (lib.getName pkg) [

“hailo10h-firmware”

“hailo-gen-ai-model-zoo”

\]*;*

```

The firmware is unavoidable: the chip does not work without it. The zoo is the

LLM server itself. Both are the price of the accelerator, and deciding the CPU

backend is the one you want is a legitimate response.

—

## 17. Diagnostic quick reference

```fish

# Is the HAT on the bus at all?

sudo lspci -nn | grep 1e60

# Which driver is actually running? (after a POWER CYCLE)

sudo dmesg | grep -i “Init module”

# Did the firmware load, and does the chip agree with the driver?

sudo dmesg | grep -iE “Firmware loaded|Mismatch|driver_compatible”

# Device node, and is it world-writable?

ls -la /dev/ | grep -iE “hailo|h1x”

# Can the runtime talk to the chip?

sudo hailortcli fw-control identify

# What does the server think it has?

curl -s http://:8000/hailo/v1/list # available in the zoo

curl -s http://:8000/api/tags # actually pulled

# Generate, with timing

curl -s http://:8000/api/generate \

-H “Content-Type: application/json” \

-d ‘{“model”:“qwen2.5-instruct:1.5b”,“prompt”:“Write two sentences about rain.”,“stream”:false}’ \

| jq ‘{total_duration, eval_count}’

```

`eval_count` divided by `total_duration / 1e9` is tokens per second.

### Full reset of everything the service owns

```fish

sudo systemctl stop hailo-ollama-models hailo-ollama hailo-ollama-seed

sudo rm -rf /var/lib/hailo-ollama/.local \

        /var/lib/hailo-ollama/hailo-ollama \\

        /var/lib/hailo-ollama/.zoo-version

sudo systemctl restart hailo-ollama-seed hailo-ollama

sleep 5

curl -s http://127.0.0.1:8000/hailo/v1/list

```

—

## 18. Files

| path | owns |

|—|—|

| `hosts/oracle.nix` | the PCIe overlay, `roles.llm` settings, `allowUnfreePredicate` |

| `modules/roles.nix` | `roles.llm.{enable,backends,cpuModels,hailoModels,host,openFirewallFor}` |

| `modules/system/llm.nix` | the `cpu` backend: `services.ollama` on 11434 |

| `modules/system/hailo.nix` | the `hailo` backend: units, firewall, assertions |

| `modules/system/hailo/driver.nix` | the kernel module and its udev rule |

| `modules/system/hailo/firmware.nix` | the blobs |

| `modules/system/hailo/runtime.nix` | `libhailort`, `hailortcli` |

| `modules/system/hailo/gen-ai-zoo.nix` | `hailo-ollama` and the model manifests |

—

## 19. Still open

- **`pci_ep`** (§6). Everything else is downstream of this.

- **Impermanence on the Pi.** The row has none, deliberately: it boots through

U-Boot and extlinux rather than systemd-boot, so the rollback module’s initrd

ordering is untested there.

- **ZFS in the closure.** `zfs-kernel` modules appear in this row’s

`extraModulePackages` despite the fleet having removed ZFS. Something is

pulling `boot.supportedFilesystems`; dead weight and a kernel-module build that

has to succeed on every bump.

- **A routing script** so calling either endpoint does not mean remembering

which port has which model.

Any humans here reading this, trying to get this working? I’m building a lot using this and I want to unleash the power.