Pipelines using USB camera are running with very low FPS

General
, , ,
1

When using a USB camera as an input a pipeline which is running at a high frame rate is running on a low frame rate 10-15 fps.
What am I doing wrong?

2

What is the problem?

Most USB cameras are not able to run at FPS larger than 15 fps with YUY2 output for resolutions larger than VGA. This is due to limited bandwidth on the USB interface. Running at high resolution with high FPS on raw video adds up quickly to high bandwidth requirements.

How to check my camera’s available modes?

There are multiple tools available to get the available modes of a camera on Ubuntu. Here are some methods you can use:

  1. Using GStreamer
gst-device-monitor-1.0

This command will list all multimedia devices and their capabilities. Look for your camera and its supported modes. It can be installed using:

sudo apt-get install gstreamer1.0-tools
  1. Using v4l2-ctl (Video4Linux2 Control)
v4l2-ctl --list-formats-ext -d /dev/video0

This tool can be installed using:

sudo apt-get install v4l-utils

Here is an example output: (Of my laptop’s built-in camera…)

v4l2-ctl --list-formats-ext -d /dev/video2
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'MJPG' (Motion-JPEG, compressed)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 320x180
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 160x120
			Interval: Discrete 0.033s (30.000 fps)

You can see that trying to get 720p raw YUYV output will be limited to 10fps.

How to use the ‘MJPG’ mode in GStreamer?

To use the ‘MJPG’ mode in GStreamer, you can add a caps filter requiring a specific video encoding.

Let’s take a simple pipeline running from the camera:

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw, width=1280, height=720 ! \
videoconvert ! fpsdisplaysink

This pipeline will run at 10 fps with the camera we are testing.

To “force” the camera to use ‘MJPG’, we’ll add specific caps:

gst-launch-1.0 v4l2src device=/dev/video2 ! image/jpeg, width=1280, height=720 ! \
decodebin ! videoconvert ! fpsdisplaysink

Note that 2 changes were required:

  1. Add the image/jpeg format caps term.
  2. Now, we need to add a decode stage in order to use the video. In this case, I used decodebin autoplugger; you can also use a specific plugin like jpegdec, for example.

Is there a downside?

Well, yes. Using an encoded stream requires decoding resources on the host. If the host is very weak, the “encoding” might become a worse bottleneck than the USB bandwidth, leading to a worse frame rate. However, in most hosts, this will allow you to reach higher frame rates than using the camera’s raw video mode.

If you found this (or other posts) useful, please hit the :heart: button to help promote it to more people. Have insights, corrections, or questions? Share your thoughts in the comments!