Should i use FPS from batch size 1 or 8?

General Discussion
1

doing a benchmark to compare performance between hailo and other devices. for FPS, is it more accurate to use batch size 1, or 8?

2

Hi @Rowan_Chin,

I’d recommend the following:

  1. Use batch size 1 - Hailo’s official FPS numbers are measured at batch size 1 with single-context HEFs compiled in performance mode, and typically this is the common figure.
  2. Make sure your HEF is from the Hailo Model Zoo - these are compiled with performance optimization enabled.
  3. If the other devices also support batched inference, consider reporting both batch 1 and batch 8 - so you can see latency-optimized vs. throughput-optimized scenarios. Note that on Hailo, --batch-size controls how many frames are processed before a context switch - it’s not traditional batch inference like on a GPU. You can check if your model uses multiple contexts with hailortcli parse-hef your_model.hef
  4. Use hailortcli benchmark your_model.hef for the easiest way to get both HW-only and full-system FPS in a single run.

Thanks,