Title: HAILO_TRACE=scheduler produces empty .hrtt file on Hailo-10H (HailoRT 5.3.0) — is scheduler-level tracing supported on H10?

jungmin_jeon · August 25, 2026, 4:54pm

Environment:

Device: Hailo-10H, PCIe (pci/0001:01:00.0)
Host: Raspberry Pi 5, aarch64, kernel 6.12.75+rpt-rpi-2712
HailoRT + hailort-pcie-driver: 5.3.0 (fresh install, hailortcli fw-control identify confirms firmware 5.3.0)
App: custom C++ benchmark using VDevice::create() with multiple InferModels (det/seg/pose, yolov8s family) sharing one VDevice so HailoRT’s built-in scheduler (scheduling_algorithm: ROUND_ROBIN, multi_process_service: false) time-slices between them — no priority/threshold/timeout params touched, all defaults.

What I’m trying to do:
I previously ran the same kind of multi-model round-robin experiment on Hailo-8L (RPi5 + AI Kit) and got fully populated .hrtt files with ActivateCoreOp/SwitchCoreOp/frame enqueue/dequeue events, using:
export HAILO_TRACE=scheduler
export HAILO_TRACE_TIME_IN_SECONDS_BOUNDED_DUMP=30
export HAILO_TRACE_PATH=/path/to/traces
export HAILO_MONITOR=1
I’m trying to do the same thing on Hailo-10H to compare scheduler behavior across chip generations.

What happens:
The .hrtt file is created, but it is always 0 bytes, regardless of:

Bounded-dump window (tried 10s and 30s)
Total process runtime (tried single-model ~15s and det+seg+pose combined ~23–25s, both longer than the bounded-dump window)
HAILO_TRACE=scheduler vs HAILO_TRACE=all (with all, no .hrtt file is even created, so scheduler does seem to be the recognized value)

I confirmed with strace -f -e trace=openat,write,close that the profiler thread does this and nothing else:
openat(AT_FDCWD, “…/hailort_.hrtt”, O_WRONLY|O_CREAT|O_TRUNC, 0666) = 7
close(7) = 0
+++ exited with 0 +++
No write() call ever happens on that fd. So this isn’t a timing/buffer-size issue — no trace events are being recorded at all, even though the vdevice log confirms scheduling_algorithm: ROUND_ROBIN is active and multi_process_service: false (so this isn’t the known MPS/HRTT issue either).

Also noticed: hailortcli monitor --help in 5.3.0 says:

▎ “On H15, presents information about the running networks (requires environment variable ‘HAILO_MONITOR’ set to 1)… On H10, presents performance and health stats.”

This suggests H10 might intentionally expose a different, coarser monitoring path than H8/H8L/H15, but I couldn’t find documentation confirming whether the scheduler-level HAILO_TRACE=scheduler HRTT mechanism specifically is supported for H10 at all.

Question: Is HAILO_TRACE=scheduler / HRTT scheduler-event tracing currently supported for Hailo-10H in HailoRT 5.3.0? If not, is there a roadmap for it, or an alternative way to inspect per-network scheduler activation/switch events on H10 (short of the coarse hailortcli monitor stats)?

Happy to share the full strace log or a minimal repro if useful.

system · August 25, 2026, 4:54pm

The Hailo Community currently supports our AI Accelerator products only.
For AI Vision Processors questions, please fill out the product inquiry form, and our team will get back to you shortly. Hailo AI Product Inquiry: Connect with Our AI Experts

jungmin_jeon · August 26, 2026, 2:53am

Thanks for the reply, but I think this may have been auto-routed incorrectly — Hailo-10H is listed under Hailo’s AI Accelerator product line (like Hailo-8/8L), not the AI Vision Processor line (Hailo-15/18 SoC family with ISP). My question isn’t about vision-processor/ISP hardware at all — it’s a pure software/firmware behavior question about HailoRT 5.3.0’s HAILO_TRACE tracing mechanism on an AI Accelerator device, which is exactly the kind of question this Community forum is meant to support.
▎
▎ To restate the core issue without the hardware framing: on Hailo-8L, HAILO_TRACE=scheduler produces a fully populated .hrtt with real scheduler events (ActivateCoreOp/SwitchCoreOp/enqueue/dequeue). On Hailo-10H, the exact same env vars and code path cause the profiler thread to open() + immediately close() the .hrtt file with zero write() calls ever occurring (confirmed via strace -f -e trace=openat,write,close), even though scheduling_algorithm: ROUND_ROBIN is confirmed active in the vdevice log. This means no scheduler trace events are being emitted internally for H10 at all — it’s not a buffer/timing issue, it’s that the instrumentation points don’t appear to fire on this device.
▎
▎ Given hailortcli monitor --help in 5.3.0 already documents a H10-specific behavior difference (“On H10, presents performance and health stats” vs. per-network info on H15), I suspect this might be a known, intentional gap for HAILO_TRACE=scheduler on H10 rather than a bug — but I haven’t found this documented anywhere, and I’d like an engineer to confirm one way or the other (supported/unsupported/roadmap) rather than have this routed to a hardware inquiry form.
▎
▎ Happy to attach the full strace log and a minimal repro if that helps get this in front of the right person. Could someone from the HailoRT team take a look?