Hardware: Intel 14th Gen i9 Host | ICY DOCK MB873MP-B V2 (8-bay OCuLink) | 8x Hailo-8 M.2 (208 TOPS) Stack: Opus (C-based) + HailoRT | XTTS v2 Model Partitioning Goal: High-density, real-time audio inference for local-first AI, routed via Tailscale to mobile clients.
Q: Are there pre-existing Hailo-optimized kernels for audio resampling or should I keep that on I9 P-cores?
Q: For a 2GB XTTS v2 model, is it more efficent to run 9 independent instances (one per NPU) or partition the encoder/decoder/vocoder across the Hailo-8 Cluster?
Hi @Michael_James,
The Hailo-8 NPU is optimized for neural network layer operations (convolutions, matrix multiplications, etc.).
As for the 2GB XTTS v2 model, the first challenge you might face is that Hailo-8 runs compiled HEF files (quantized INT8/INT16 models processed through the Hailo Dataflow Compiler), so XTTS v2 would need to go through that compilation flow - and not all operations may be supported, which is worth verifying first.
If compilation succeeds, HailoRT does support multi-device inference via the VDevice API - this transparently distributes the same compiled model across multiple devices for higher throughput.
So for the XTTS v2 use case, running independent instances (one per NPU) might be the more practical path to explore, assuming each submodel (encoder/decoder/vocoder) can be compiled to a HEF individually.
Thanks,
Michael.