So I’m building a little device that plugs into my laptop and uses object detection to automatically aim at certain objects on the screen. The input is fed from my laptop’s HDMI port to the Pi 5 via a USB capture card. A python script calculates the distance between the center of the bounding box and the center of the screen. A Pi Pico with a USB HID library acts as the mouse. If the object is not in the center, the Pi 5 sends 3v to the appropriate pins on the Pico to move the mouse up, down, left, and right at a constant speed. The problem is that the mouse starts oscillating and orbiting the target instead of locking onto it. I’m guessing this is because of latency. There’s an obvious delay between what’s shown on the screen and what the AI sees, which is causing it to overshoot its target but not notice it until the video feed catches up. This then causes it swing back and overshoot its target again, making it freak out. This becomes obvious when I set the mouse speed on the Pico to a very slow speed. Then it becomes good at keeping the object centered. But if I slow it down, then I can’t use it for its intended purpose of shooting at fast-moving targets. Any ideas on solving the latency?
Hey @abramnick2003 ,
I would skip some frames after i get the first positive and would calculate the maximum it takes to move the mouse to that and will skip that amount of frames.