Sidebar
Table of Contents
Stereoscopic recording using FFmpeg
This is I support material collected for Steroscopic/VR workshops by Radiona. We are using two identical USB cameras with FFmpeg for capturing video or images. FFmpeg is a free software project that produces libraries and programs for handling multimedia data available for most operating systems (Windows, Linux, Mac OS X, Android…)
Accessing USB cameras
Examples here are tested on Windows (DirectShow multimedia framework), but with small changes it should work on other systems. https://trac.ffmpeg.org/wiki/Capture/Webcam. Linux/Debian v4l-utils package.
List dshow input devices (Windows):
ffmpeg -list_devices true -f dshow -i dummy
Linux:
v4l2-ctl --list-devices
Print a list of options from a selected device:
ffmpeg -f dshow -list_options true -i video="Icatch(X) KV Series Null Device"
Linux:
ffmpeg -f v4l2 -list_formats all -i /dev/video0
Camera calibration
Preview/overlay both cameras at the same time for position alignment (there is significant lag but still usable). Adjust for your camera name and supported formats & color modes.
ffmpeg -y -f dshow -video_size 640x480 -framerate 30 -pixel_format yuyv422 -video_device_number 0 -i video="Icatch(X) KV Series Null Device" -f dshow -video_size 640x480 -framerate 30 -pixel_format yuyv422 -video_device_number 1 -i video="Icatch(X) KV Series Null Device" -filter_complex blend=all_expr='if(eq(mod(X,2),mod(Y,2)),A,B)' -vcodec rawvideo -pix_fmt yuyv422 -f sdl "SDL output"
Video recording
Parallel video recording of two cameras into separate files. Before recording find-out which one is right and left (Usually corresponds to order of plugging them).
ffmpeg -y -f dshow -video_size 640x480 -framerate 30 -pixel_format yuyv422 -video_device_number 0 -i video="Icatch(X) KV Series Null Device" LEFT.mp4 | ffmpeg -y -f dshow -video_size 640x480 -framerate 30 -pixel_format yuyv422 -video_device_number 1 -i video="Icatch(X) KV Series Null Device" RIGHT.mp4
Record direct mjpeg stream – bigger resolution and without lag.
ffmpeg -y -f dshow -video_size 1280x720 -framerate 30 -vcodec mjpeg -video_device_number 0 -i video="Icatch(X) KV Series Null Device" -vcodec copy RIGHT.mp4 | ffmpeg -y -f dshow -video_size 1280x720 -framerate 30 -vcodec mjpeg -video_device_number 1 -i video="Icatch(X) KV Series Null Device" -vcodec copy LEFT.mp4
Rotating the videos
In case you are recording with rotated cameras, we need to unrotate the video (in this case 90° CW).
ffmpeg -i LEFT.mp4 -codec:v libx264 -profile:v high -preset slower -b:v 1000k -vf "transpose=1" -threads 0 -codec:a aac -b:a 196k LEFT_rot.mp4
Framepacking in stereo video
Pack two different video streams into a stereoscopic video, setting proper metadata on supported codecs. Convert left and right video to one side-by-side (SBS). https://ffmpeg.org/ffmpeg-filters.html#framepack
ffmpeg -i LEFT.mp4 -i RIGHT.mp4 -filter_complex framepack=sbs OUTPUT_SBS.mp4
Convert views into a side-by-side video with the same output resolution as the input (SBS half resolution)
ffmpeg -i LEFT.mp4 -i RIGHT.mp4 -filter_complex [0:v]scale=w=iw/2[left],[1:v]scale=w=iw/2[right],[left][right]framepack=sbs OUTPUT_SBS.mp4
Convert views into a top-above-bottom format:
ffmpeg -i LEFT.mp4 -i RIGHT.mp4 -filter_complex framepack=tab OUTPUT_TAB.mp4
Framepacking might not work with files recorded with direct stream so we need to recompress it forcing the framerate.
ffmpeg -i LEFT.mp4 -r 30 -vcodec libx264 -preset ultrafast -qp 0 LEFT_2.mp4 ffmpeg -i RIGHT.mp4 -r 30 -vcodec libx264 -preset ultrafast -qp 0 RIGHT_2.mp4
Metadata for Youtube video uploads
Unfortunately none of the below recommended metadata work and Youtube is breaking things constantly. It might recognize stereoscopic/3D video in the computer browser (only anaglyph option) but it won't give the cardboard options on Youtube on smartphone. Additionally aspect ratio might be interpreted differently between computer and smartphone. It seems the best way for now is to prerender everything and bypass Youtube's unreliable/crappy 3d support. 
MPEG4 (with re-encoding):
ffmpeg -i input_file.mp4 -vcodec libx264 -x264opts frame-packing=3 output_file.mp4
ffmpeg -i input_file.mkv -vcodec libx264 -x264opts "frame-packing=3:frame-packing-interpret=1:frame-packing-quincunx=0:frame-packing-grid=0,0,0,0" output_file.mp4
Matroska (direct stream copy)
ffmpeg -i input_file.mkv -c copy -metadata:s:v:0 stereo_mode=1 output_file.mkv
There is Google's tool for metadata injection but it also didn't work in this case. https://github.com/google/spatial-media/ This fork is less strict and allows more operations: https://github.com/Vargol/spatial-media
Add left-right (SBS) metadata:
python spatialmedia -i --stereo=left-right INPUT.mp4 OUTPUT.mp4
Print spatial media metadata contained in the file:
python spatialmedia FILE
Changing stereoscopic format
Convert between different stereoscopic image formats. https://trac.ffmpeg.org/wiki/Stereoscopic https://ffmpeg.org/ffmpeg-filters.html#stereo3d
Converting SBS (full res) to anaglyph (Red-cyan dubois):
ffmpeg -i OUTPUT_SBS.mp4 -vf stereo3d=sbsl:arcd OUTPUT_ANAGLYPH.mp4
Still image capture
Direct stereo SBS image capture from usb cams. There is lattency between left and rignt camera.
ffmpeg.exe -y -f dshow -video_size 1600x1200 -vcodec mjpeg -video_device_number 0 -i video="Icatch(X) KV Series Null Device" -f dshow -video_size 1600x1200 -vcodec mjpeg -video_device_number 1 -i video="Icatch(X) KV Series Null Device" -filter_complex framepack=sbs -vframes 1 STEREO_SBS.jpeg
SBS image to ANAGLYPH image
ffmpeg -i STEREO_SBS.jpeg -vf stereo3d=sbsl:arcd -vframes 1 STEREO_ANAGLYPH.jpeg
