Hi everyone,
I am a graduate student currently researching scheduling on the Hailo-8. I’m posting to see if anyone has insights or empirical data regarding the context switching latency during model transitions on the Hailo-10H.
As we know, the Hailo-8 lacks internal DRAM, meaning that every model switch requires reloading weights entirely over the PCIe interface. On the other hand, the Hailo-10H features internal LPDDR, which should theoretically reduce this transition latency significantly. However, I am looking for concrete, measured latency data to see just how much it actually improves in practice.
For reference, on the Hailo-8, my measurements for Model Zoo networks show a context switching latency of about 5.64ms for resnet_v1_50 and 4.93ms for yolov8s. This aligns closely with the maximum bandwidth of a PCIe Gen3 x4 connection. Furthermore, when deployed on platforms with limited PCIe performance like the Raspberry Pi 5 (PCIe Gen2 x1), this latency degrades even further.
While the Hailo-10H’s internal LPDDR should drastically cut down these numbers, I want to verify if that is truly the case in real-world testing, or if there are other architectural overheads that keep the latency relatively high.
If anyone has benchmarked or measured these specific context switching latencies on the Hailo-10H, could you please share your findings?
Thank you!