Multicamera inference

General
,
1

Hello,
I created this code to perform inference on frames from two Arducam cameras. I’m running inference in batches of two images—one from each camera. The code works well, but before running inference, I’m getting 54 FPS. After performing inference on the two images, the frame rate drops to 28 FPS.
My model is a custom YOLOv8n with two classes, and I’m using a Raspberry Pi CM4. What do you suggest I can do to increase the inference rate?
Thanks!

from picamera2 import Picamera2
from picamera2.devices import Hailo
import numpy as np
import time
import cv2


def extract_detections(hailo_output, w, h, class_names, cam_id, threshold=0.5):
    """Extract detections from the HailoRT-postprocess output."""
    results = []
    for class_id, detections in enumerate(hailo_output):
        for detection in detections:
            detection_array = np.array(detection)
            score = detection_array[4]
            if score >= threshold:
                y0, x0, y1, x1 = detection_array[:4]
                bbox = (int(x0 * w), int(y0 * h), int(x1 * w), int(y1 * h))
                results.append([class_names[class_id], bbox, score])
                print(
                    f"Detection(s) found for class '{class_names[class_id]}', Score: {score:.2f}, Camera: {cam_id}"
                )
    return results


if __name__ == "__main__":

    video_w, video_h = 1920, 1080
    model = "yolov8n_D4.hef"
    labels = "labels.txt"
    score_thresh = 0.5

    hailo = Hailo(model, 2)
    model_h, model_w, _ = hailo.get_input_shape()

    try:
        with open(labels, "r", encoding="utf-8") as f:
            class_names = f.read().splitlines()
    except FileNotFoundError as file_error:
        print(f"Error: The labels file '{labels}' was not found.")
        raise file_error
    except IOError as io_error:
        print(f"Error: There was an issue reading the labels file '{labels}'")
        raise io_error
    except Exception as e:
        print(f"Unexpected error while loading labels from '{labels}'")
        raise e

    detections = None

    picam2a = Picamera2(0)
    picam2b = Picamera2(1)

    # Configure camera A
    main_a = {'size': (video_w, video_h), 'format': 'XRGB8888'}
    lores_a = {'size': (model_w, model_h), 'format': 'YUV420'}
    controls_a = {
        'FrameRate': 60,
        "AfMode": 0,          # Manual AF mode
        "NoiseReductionMode": 0,  # Noise reduction off
        "LensPosition": 6.0,
    }
    config_a = picam2a.create_preview_configuration(
        main_a, lores=lores_a, controls=controls_a)
    picam2a.configure(config_a)

    # Configure camera B
    main_b = {'size': (video_w, video_h), 'format': 'XRGB8888'}
    lores_b = {'size': (model_w, model_h), 'format': 'YUV420'}
    controls_b = {
        'FrameRate': 60,
        "AfMode": 0,
        "NoiseReductionMode": 0,
        "LensPosition": 6.0,
    }
    config_b = picam2b.create_preview_configuration(
        main_b, lores=lores_b, controls=controls_b)
    picam2b.configure(config_b)

    picam2a.start()
    picam2b.start()

    input_data = np.empty((2, model_h, model_w, 3), dtype=np.uint8)

    start_time = time.time()

    while True:
        frame_a = picam2a.capture_array('lores')
        frame_b = picam2b.capture_array('lores')

        rgb_a = cv2.cvtColor(frame_a, cv2.COLOR_YUV420p2RGB)
        rgb_b = cv2.cvtColor(frame_b, cv2.COLOR_YUV420p2RGB)

        input_data[0] = rgb_a
        input_data[1] = rgb_b

        results = hailo.run(input_data)

        for cam_id, result in enumerate(results):
            detections = extract_detections(
                result, video_w, video_h, class_names, cam_id, score_thresh
            )
2

Hey @dgarrido,

You’re currently using blocking inference with:

hailo.run(input_data)

That means each inference call waits until the result is ready before continuing — which creates idle time and limits your FPS.

To improve performance, you should move to asynchronous inference. Hailo supports this in two main ways:

  1. Raspberry Pi Integration (Async with Picamera2 + Hailo module)
    The Raspberry Pi Hailo integration lets you run async inference natively:

    from picamera2 import Picamera2
from picamera2.devices import Hailo

picam2 = Picamera2()
picam2.start()

with Hailo("model.hef") as hailo:
    future = hailo.run_async(frame)  # non-blocking
    # do other processing...
    results = future.result()  # get the result when needed

    This approach allows you to keep your camera running while inference is being processed in the background.

  2. Native HailoRT Async API
    If you want more control, the lower-level HailoRT async API gives you full async handling. You can find an example here:
    :backhand_index_pointing_right: Hailo-Application-Code-Examples/runtime/python/utils.py at main · hailo-ai/Hailo-Application-Code-Examples · GitHub

In your current setup, capture and inference are happening one after the other — frame capture, color conversion, then inference. This serial flow means the Hailo device often sits idle, waiting.

To fix this, you should separate capture and inference into parallel threads. One thread continuously captures frames into a queue, and another thread pulls from that queue and runs inference. This keeps both the camera and Hailo busy and improves overall throughput.

Another thing to look at is image preprocessing. You’re using:

cv2.cvtColor(frame_a, cv2.COLOR_YUV420p2RGB)

This is expensive on the Pi. If your camera supports it, switch to direct RGB output (format='RGB888') to skip this step. If not, consider using OpenCV’s UMat for faster, hardware-accelerated conversion.

Also, Logging every detection like:

print(f"Detection(s) found for class '{class_names[class_id]}', Score: {score:.2f}, Camera: {cam_id}")

can slow things down on a Pi. Try throttling the logs or disabling them in your hot loop.

Lastly, you’re already using batch size 2 — which is solid. But depending on memory constraints, you might be able to push it to 4, 8, or even 16, and see a boost in FPS.