There are two points.
Your Raspberry Pi is set to PCI gen 2. Please follow this guide to upgrade it to PCIe gen 3.
How to upgrade to PCIe gen-3 in RPi5 with Hailo-8L M.2
The second point, the numbers in the Model Zoo are for a Hailo device connected to a system with 4 PCIe lanes and a x86 CPU. Your Raspberry Pi only has a single lane.
Especially for multi-context networks PCIe bandwidth will influence the maximum FPS you can reach.