Hailo-10H / HailoRT 5.1.1 — KV-cache state retention across repeated InferModel requests

General Discussion
,
1

Title: Hailo-10H / HailoRT 5.1.1 — KV-cache state retention across repeated InferModel requests

Hello Hailo Community,

We’re investigating numerical reproducibility when reusing a VDevice for sequential transformer inference on a Raspberry Pi 5 with a Hailo-10H accelerator.

Our environment uses HailoRT and firmware 5.1.1, with a custom transformer HEF containing separate prefill and token-by-token inference groups and a 128-position KV cache.

We’ve observed the following:

  1. Repeated identical requests using a shared VDevice can produce different numerical outputs, even when the input buffers, attention masks, and positional encodings are identical.

  2. Recreating the VDevice for each request restores bitwise reproducibility in our tested sequences.

  3. Repeated prefill-only inference also exhibits the discrepancy.

  4. Explicitly restoring the cache offset to zero between prefill-only requests restores bitwise-identical results.

  5. Applying the equivalent offset restoration after full autoregressive generation improves reproducibility but does not eliminate numerical differences.

We’ve examined the public HailoRT source between versions 5.1.1 and 5.4.0 and noticed changes involving cache-buffer metadata, cache IDs, and GenAI context management.

Our questions:

  • Does HailoRT require explicit KV-cache initialization when beginning a new independent request on an existing VDevice?

  • Is restoring the cache offset sufficient, or must cache contents or descriptors also be reinitialized?

  • Were any changes in HailoRT 5.2.0, 5.3.0, or 5.4.0 intended to address cache lifecycle management, request isolation, or numerical reproducibility?

  • Is there a documented or recommended way to reset the KV cache through the generic InferModel API, rather than the GenAI LLM interface?

We are specifically interested in the supported lifecycle contract and any relevant changelog entries, rather than installation or upgrade instructions.

We have reproducible numerical comparisons and can provide a minimal, non-proprietary reproduction if useful.

Thank you for any guidance.

James Childers
AIToolworx | dAIko Development