Hi everyone,
I’ve been experimenting with moving the computationally heavy backend of Piper/VITS TTS onto the Hailo-10H, targeting a Raspberry Pi 5 + AI HAT+ 2.
This is still a proof of concept and has not yet been validated on physical Hailo-10H hardware. So far, the backend has been tested using the Hailo SDK emulator, and both networks have successfully compiled into Hailo-10H HEFs.
The current split is:
Raspberry Pi CPU
text → phonemes → encoder → duration/alignment → latent + mask
Hailo-10H
latent + mask
→ INT8 VITS reverse flow
→ host-side mask multiply
→ INT8 waveform decoder
→ 22.05 kHz audio
The current voice is Piper Amy Medium.
I did use AI assistance heavily for this project. I wanted to see if I could get TTS running on the Hailo-10H, and I wasn’t seeing much work in this area.
One project that caught my attention was:
https://huggingface.co/cstr/all-MiniLM-L6-v2-hailo10h
That work was more focused on the encoder/front-end side of a different TTS/model stack, while I decided to concentrate on the synthesis backend of Piper/VITS using Amy Medium.
Some of the main things I worked through with AI assistance and a lot of experimentation were:
-
fixed accelerator length of
T=148 -
persistent 4D tensor layout
-
conversion of the Conv1D/ConvTranspose1D decoder into equivalent Conv2D/ConvTranspose2D operations
-
graph surgery on several reverse-flow subtraction operations after finding a quantized sign issue
-
calibration using real Amy internal activations
-
decoder QAT using a 1024-entry real-flow-manifold calibration set
The FP32 4D decoder rewrite matched the original decoder essentially exactly before quantization.
The current target path:
INT8 flow → INT8 decoder
produces intelligible speech in the SDK emulator for the held-out phrase:
"This is Amy."
Current emulator metrics over the valid speech region:
correlation vs FP32 : 0.9411
RMSE : 0.0307
SNR : 9.28 dB
quiet floor : -49.1 dBFS
There is still audible decoder quantization noise, so I consider this POC quality rather than finished TTS.
Final compiled backend:
Reverse flow:
INT8
3 contexts
~7.23 MiB HEF
Decoder:
INT8
2 contexts
~2.21 MiB HEF
Accelerator tensors:
latent: [1, 148, 1, 192]
mask: [1, 148, 1, 1]
One useful lesson from this work has been that keeping audio/sequence tensors in a stable 4D representation seems much easier for the Hailo toolchain than repeatedly moving between 3D and 4D forms.
I’ve put the deployment scripts, HEF information, test vectors, results, and lessons learned here:
GitHub:
https://github.com/steveonw/Piper-VITS-TTS-on-Hailo-10H
The next step is physical validation on a Raspberry Pi 5 + AI HAT+ 2. I’m particularly interested in:
-
whether both multi-context HEFs execute reliably on Hailo-10H
-
flow and decoder hardware latency
-
host/runtime overhead between the two stages
-
long-run stability
-
whether anyone sees the vDMA/completion behavior reported in some recent Hailo-10H Pi threads
I’m not hiding the fact that I used AI. I didn’t start this as a Hailo or ML compiler expert — I just wanted TTS running on the Hailo-10H and spent a solid week going through a ridiculous number of AI-assisted iterations, Google Colab sessions, compiler failures, graph changes, calibration attempts, and listening tests to get to the point where I was comfortable sharing the results.
I also kept the development history. I’m willing to share the Jupyter notebooks and intermediate files if somebody wants to reproduce or dig into what I did.
Fair warning: the development archive is around 2 GB, but it should give a pretty good picture of how the project evolved.
If anyone with an AI HAT+ 2 is interested in testing it, I’d really appreciate results or feedback.
Thanks — and I’m very open to corrections if I’m doing something weird with HailoRT or the compiler.