Hailo-8 AI HAT+ clock rate locked at 400 MHz after initial 800 MHz operation (Pi 5 + Ubuntu 24.04)

General Discussion
,
1

Hello Everyone,

A Hailo-8 AI HAT+ (26 TOPS variant, Raspberry Pi-branded) was initially measured running at 800 MHz with 398 FPS hw_only / 6.71 ms latency on YOLOv8s. After a system reboot, the chip transitioned to a 400 MHz state that has persisted across multiple cold power cycles, full reboots, driver reloads, and firmware re-uploads. Performance dropped to ~38 FPS hw_only / ~32 ms latency.

The 5 V rail is healthy (5.13 V), PCIe is at Gen 3 8 GT/s, the HEF is the correct single-context YOLOv8s, and no firmware load errors appear in dmesg. The chip appears physically and electrically fine; the issue is firmware/PM-state level.

Looking for guidance on whether this is a recoverable state, what the changing PM Values indicate, and whether there is a firmware reset sequence that returns the chip to its initial 800 MHz operating point.

Hardware

  • Host: Raspberry Pi 5 (8 GB RAM, official board)
  • Accelerator: Raspberry Pi AI HAT+ (26 TOPS Hailo-8 variant)
  • Power supply: Official Raspberry Pi 27 W USB-C PSU
  • Carrier: Pi 5 PCIe FFC ribbon connector → AI HAT+ stacking header
  • Cooling: Passive heatsink only (NPU never exceeded 51.8 °C peak during testing)

Software

  • OS: Ubuntu 24.04 Server (aarch64)
  • Kernel: 6.8.0-1052-raspi
  • HailoRT driver: v4.20.0 (built from source against running kernel)
  • HailoRT library: v4.20.0 (matched to driver tag)
  • Firmware: 4.20.0 (release,app,extended context switch buffer)
  • HEF: YOLOv8s from model-zoo v2.14.0/hailo8/yolov8s.hef, verified single-context

Timeline

Initial state — 800 MHz, full performance (one session only):

hailortcli benchmark resources/yolov8s.hef
=======
Summary
=======
FPS     (hw_only)                 = 398.43
        (streaming)               = 318.27
Latency (hw)                      = 6.71 ms

hailortcli run resources/yolov8s.hef --input-files <video_frames.bin> \
    --frames-count 336 --measure-latency --measure-overall-latency
HW Latency:      6.71 ms
Overall Latency: 11.46 ms

hailortcli fw-control identify --extended during this session reported Neural Network Core Clock Rate: 800MHz (output not captured at the time — the issue was discovered only after the transition).

After first reboot — 400 MHz, persistent:

hailortcli benchmark resources/yolov8s.hef
=======
Summary
=======
FPS     (hw_only)                 = 38.57
        (streaming)               = 32.51
Latency (hw)                      = 32.48 ms

The chip has remained in this state across all subsequent recovery attempts.

Full diagnostic output

hailortcli fw-control identify --extended

Executing on device: 0000:01:00.0
Identifying board
Control Protocol Version: 2
Firmware Version: 4.20.0 (release,app,extended context switch buffer)
Logger Version: 0
Board Name: Hailo-8
Device Architecture: HAILO8
Serial Number: <N/A>
Part Number: <N/A>
Product Name: <N/A>
Boot source: PCIE
Neural Network Core Clock Rate: 400MHz
Device supported features: PCIE
LCS: 3
SoC ID: 626EC929B00364C1560F6D8C077D3D135782DB7941249677141C35EABEDFCE69
ULT ID: 0060C1C369B35850FB7A9544

PM Values observed across runs

The PM Values change between runs even though the reported clock stays at 400 MHz:

Run 1 (after first transition):       0246010000029B010000021C020000E9031142509F114201
Run 2 (after driver reload):          024501000002990100000219020000293633428FD1334201
Run 3 (after full reboot):            024501000002990100000219020000293633428FD1334201
Run 4 (after cold power cycle):       0245010000029A010000021A02000022032242899E224201

The first 10 hex digits appear to differ between sessions. Hoping someone can decode these — the documentation does not appear to cover this field.

dmesg (Hailo driver probe + firmware load)

[    6.937062] hailo 0000:01:00.0: Probing: Device enabled
[    6.937140] hailo 0000:01:00.0: Probing: mapped bar 0 - 000000009d546ddb 16384
[    6.937146] hailo 0000:01:00.0: Probing: mapped bar 2 - 000000005b89717d 4096
[    6.937150] hailo 0000:01:00.0: Probing: mapped bar 4 - 00000000a420c170 16384
[    6.937219] hailo 0000:01:00.0: Probing: Setting max_desc_page_size to 4096, (page_size=4096)
[    6.937230] hailo 0000:01:00.0: Probing: Enabled 64 bit dma
[    6.937234] hailo 0000:01:00.0: Probing: Using userspace allocated vdma buffers
[    6.937263] hailo 0000:01:00.0: Disabling ASPM L0s
[    6.937287] hailo 0000:01:00.0: Successfully disabled ASPM L0s
[    6.945744] hailo 0000:01:00.0: Writing file hailo/hailo8_fw.bin
[    7.534953] hailo 0000:01:00.0: File hailo/hailo8_fw.bin written successfully
[    7.534961] hailo 0000:01:00.0: Writing file hailo/hailo8_board_cfg.bin
[    7.536433] Failed to write file hailo/hailo8_board_cfg.bin
[    7.536441] hailo 0000:01:00.0: File hailo/hailo8_board_cfg.bin written successfully
[    7.536444] hailo 0000:01:00.0: Writing file hailo/hailo8_fw_cfg.bin
[    7.536494] Failed to write file hailo/hailo8_fw_cfg.bin
[    7.536496] hailo 0000:01:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[    7.677156] hailo 0000:01:00.0: NNC Firmware loaded successfully
[    7.677165] hailo 0000:01:00.0: FW loaded, took 731 ms
[    7.690379] hailo 0000:01:00.0: Probing: Added board 1e60-2864, /dev/hailo0

The Failed to write file hailo/hailo8_board_cfg.bin and Failed to write file hailo/hailo8_fw_cfg.bin lines stand out — they appear immediately before written successfully messages for the same files. These files do not exist in /lib/firmware/hailo/ on the system; only hailo8_fw.bin was installed via the standard install procedure. It is unclear whether these are expected failures (driver attempts an optional config load) or whether missing files are contributing to the clock state.

PCIe link status

$ sudo lspci -vv -s 01:00.0 | grep -iE "lnksta|lnkcap"
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
LnkSta: Speed 8GT/s, Width x1 (downgraded)
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+

PCIe Gen 3 8 GT/s is active. The (downgraded) annotation is the expected Pi 5 behavior (the controller advertises Gen 4 while negotiating Gen 3).

Pi 5 power rail readings

$ vcgencmd pmic_read_adc | grep -iE "5v|3v"
3V7_WL_SW_A current(0)=0.05855580A
3V3_SYS_A   current(1)=0.13565430A
3V3_DAC_A   current(17)=0.00000000A
3V3_ADC_A   current(18)=0.00000000A
3V7_WL_SW_V volt(8)=3.71462400V
3V3_SYS_V   volt(9)=3.31374500V
3V3_DAC_V   volt(20)=3.31593100V
3V3_ADC_V   volt(21)=3.31959400V
EXT5V_V     volt(24)=5.13488000V

5 V input rail at 5.13 V — comfortably above the 4.8 V undervoltage floor.

Chip temperature during inference (400 MHz state)

$ hailortcli run resources/yolov8s.hef --frames-count 100 --measure-temp
...
Minimum chip temperature: 45.6376 C
Average chip temperature: 45.9603 C
Maximum chip temperature: 46.1687 C

Chip is running cool — never exceeded 46 C during inference in the 400 MHz state.

/boot/firmware/config.txt relevant entries

dtparam=pciex1_gen=3
usb_max_current_enable=1

Both lines present and confirmed effective (Gen 3 link is active).

Recovery attempts that did NOT restore 800 MHz

  1. sudo modprobe -r hailo_pci && sudo modprobe hailo_pci (driver reload, firmware re-uploaded in 366 ms)
  2. sudo reboot (full software reboot)
  3. Full system shutdown, PSU disconnected for 30+ seconds, cold power-on
  4. Re-running download_firmware.sh and reinstalling /lib/firmware/hailo/hailo8_fw.bin
  5. Driver rebuild from hailort-drivers at tag v4.20.0

In every case, post-recovery fw-control identify --extended reports Neural Network Core Clock Rate: 400MHz.

Verification that the HEF is correct

$ hailortcli parse-hef resources/yolov8s.hef | head -10
Architecture HEF was compiled for: HAILO8
Network group name: yolov8s, Single Context
    Network name: yolov8s/yolov8s
        VStream infos:
            Input  yolov8s/input_layer1 UINT8, NHWC(640x640x3)
            Output yolov8s/yolov8_nms_postprocess FLOAT32, HAILO NMS BY CLASS(...)

$ ls -lh resources/yolov8s.hef
-rw-rw-r-- 1 user user 11M resources/yolov8s.hef

$ md5sum resources/yolov8s.hef
1d53dcd1a454f5ebcd0fc94d2112ccf3  resources/yolov8s.hef

Compiled for HAILO8, single-context, 11 MB. Same file that produced the original 398 FPS result.

Questions for the community

  1. Is the reported Neural Network Core Clock Rate: 400MHz a recoverable state, or does it indicate that this chip is factory-provisioned at 400 MHz? Community posts suggest 400 MHz is associated with a reduced-power Hailo-8 variant, but this hardware was sold as a 26 TOPS AI HAT+ and produced 398 FPS at one point during testing, which is consistent with full 800 MHz operation.
  2. Are there documented firmware reset sequences that can return the chip to its initial clock state? Standard driver reload and PCIe link reset have been attempted without effect.
  3. Can someone with internal access decode the PM Values strings shown above? The values change between runs even at the same reported clock rate, suggesting power-state activity that does not surface in the textual output.
  4. Are the Failed to write file hailo/hailo8_board_cfg.bin and Failed to write file hailo/hailo8_fw_cfg.bin dmesg lines related? The files do not exist in /lib/firmware/hailo/ — are they supposed to be installed, and if so, would installing them affect the clock state?
  5. Is there a way to query the chip for its provisioned hardware variant (i.e. to distinguish a defective full-clock Hailo-8 stuck at half-clock vs a factory-provisioned half-clock variant that happened to run at full clock once)?

The initial 398 FPS / 6.71 ms result is reproducible-quality data, captured under the same conditions as the current 38 FPS / 32 ms results. If the chip is genuinely meant to run at 400 MHz, the original measurement is an anomaly that itself needs an explanation. If the chip is meant to run at 800 MHz and is now stuck at 400 MHz, recovery guidance would be much appreciated.

Happy to provide additional diagnostic output, run specific test commands, or share the full benchmark suite results from both clock states.

Thank you.

2

Hi @Joao_Martins,

In Hailo-8, the clock is always 400MHz, It’s physically impossible to reach 800MHz.
Maybe it’s confusing 800 with something else?

Thanks,

3

Hi Michael,

Thanks for the clarification, that resolves the confusion on my end about the clock spec.
Accepting that 400 MHz is the maximum and 800 MHz was never a real operating state, I’d like to understand two follow-up points:

1. What is the Neural Network Core Clock Rate field actually reporting? Across the original session, the output of hailortcli fw-control identify --extended showed Neural Network Core Clock Rate: 800MHz, and the same command now consistently shows 400MHz. If 800 MHz is physically impossible, what state or value would have caused the firmware to report it? Is this field documented anywhere, or is it perhaps a derived/calculated value rather than the actual core clock?

2. The performance gap is real and unexplained. Whatever the clock readout meant, the measured throughput on the same hardware, same HEF, same install dropped from 398 FPS hw_only / 6.71 ms latency to ~38 FPS hw_only / ~32 ms latency — a 10× collapse, not a 2× one. Both numbers came from hailortcli benchmark on the public model-zoo v2.14.0/hailo8/yolov8s.hef. For reference, Hailo’s published datasheet figures for YOLOv8s on Hailo-8 are in the 300+ FPS range, so the original 398 FPS is consistent with the spec and the current ~38 FPS is roughly 10% of expected.

If the chip is operating at its intended 400 MHz, what would account for ~38 FPS instead of the expected ~300+ FPS on a single-context HEF? Possible angles I’d appreciate guidance on:

  • Could the PM Values field (which keeps changing between runs at 024501... / 024601...) indicate a degraded power state?
  • The dmesg lines Failed to write file hailo/hailo8_board_cfg.bin and Failed to write file hailo/hailo8_fw_cfg.bin — should those files exist in /lib/firmware/hailo/? They aren’t in the standard hailort-drivers install.
  • Is there a way to verify whether the chip is in a normal vs degraded firmware state, given the SoC ID and ULT ID?

Happy to provide any additional diagnostic output.

Thanks again for the clarification on the clock spec.
Regards,

4

Hi @Joao_Martins,

  1. It shows the max_neural_network_core_clock_rate value stored in the device’s firmware configuration (flash or EEPROM). Per the HailoRT documentation, the supported values for this field are 100, 200, and 400 MHz, and it cannot be set above the maximum supported value for each board or module. For the Hailo-8 AI HAT+, 400 MHz is the correct maximum.

  2. The Failed to write file hailo/hailo8_board_cfg.bin and hailo8_fw_cfg.bin messages in dmesg are actually expected and benign - these are optional user-config override files that don’t ship with the standard driver install, and the driver continues normally without them (you’ll notice it still logs “written successfully” right after). For the 10x performance collapse, which is far too severe to be explained by a clock change alone, you might want to first check your PCIe link speed with sudo lspci -vv and look for the Hailo device’s LnkSta: line - if it shows 2.5GT/s instead of 8GT/s, your link has fallen back to Gen2, which alone would roughly halve throughput (and on RPi5, Gen3 x1 already gives about half the Model Zoo numbers since those are measured on Gen3 x2). Beyond that, you might try a full cold power cycle of the Pi - not just a reboot but actually unplugging power for 10+ seconds - since the Hailo firmware state is loaded fresh at boot from the host driver and can sometimes end up in an odd state, and if the issue persists after confirming Gen3 link and a cold restart, it could be worth checking hailortcli fw-control identify --extended for the temperature reading to rule out thermal throttling.

Thanks,

5

Hi @Michael,

The cold power cycle unlocked the Hailo. It achieves ~390 fps again and keeps Gen3 link.
After a week running, no issues reported.

Thank you in advance for your guidance!

Best Regards,

1 Like
6

Thanks @Joao_Martins and glad to hear now it’s working!