Content-preserving warps for 3D video stabilization

ABSTRACT

We describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method adjusts the video to appear as if it were taken from nearby viewpoints, allowing 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.

REFERENCES

• 1: Marc Alexa , Daniel Cohen-Or , David Levin, As-rigid-as-possible shape interpolation, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.157-164, July 2000 [doi>10.1145/344779.344859]
• 2: Shai Avidan , Ariel Shamir, Seam carving for content-aware image resizing, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 3: Thaddeus Beier , Shawn Neely, Feature-based image metamorphosis, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.35-42, July 1992
• 4: Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Rendering Techniques 2007: 18th Eurographics Workshop on Rendering, 327--338.
• 5: F. L. Bookstein, Principal Warps: Thin-Plate Splines and the Decomposition of Deformations, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.11 n.6, p.567-585, June 1989 [doi>10.1109/34.24792]
• 6: Buehler, C., Bosse, M., and McMillan, L. 2001. Nonmetric image-based rendering for video stabilization. In 2001 Conference on Computer Vision and Pattern Recognition (CVPR 2001), 609--614.
• 7: Chris Buehler , Michael Bosse , Leonard McMillan , Steven Gortler , Michael Cohen, Unstructured lumigraph rendering, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.425-432, August 2001 [doi>10.1145/383259.383309]
• 8: Yung-Yu Chuang , Aseem Agarwala , Brian Curless , David H. Salesin , Richard Szeliski, Video matting of complex scenes, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 9: Andrew Fitzgibbon , Yonatan Wexler , Andrew Zisserman, Image-Based Rendering Using Image-Based Priors, International Journal of Computer Vision, v.63 n.2, p.141-151, July 2005 [doi>10.1007/s11263-005-6643-9]
• 10: Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 297--304.
• 11: Michael L. Gleicher , Feng Liu, Re-cinematography: Improving the camerawork of casual video, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.5 n.1, p.1-28, October 2008 [doi>10.1145/1404880.1404882]
• 12: Michael Gleicher , Andrew Witkin, Through-the-lens camera control, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.331-340, July 1992
• 13: Jonas Gomes , Lucia Darsa , Bruno Costa , Luiz Velho, Warping and morphing of graphical objects, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 14: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer vision, Cambridge University Press, New York, NY, 2000
• 15: Paul S. Heckbert, Fundamentals of Texture Mapping and Image Warping, University of California at Berkeley, Berkeley, CA, 1989
• 16: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 17: Takeo Igarashi , Tomer Moscovich , John F. Hughes, As-rigid-as-possible shape manipulation, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 18: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 19: Kawin, B. 1992. How Movies Work. Univ. of California Press.
• 20: J. Lee , S. Y. Shin, General Construction of Time-Domain Filters for Orientation Data, IEEE Transactions on Visualization and Computer Graphics, v.8 n.2, p.119-128, April 2002 [doi>10.1109/2945.998665]
• 21: Yasuyuki Matsushita , Eyal Ofek , Weina Ge , Xiaoou Tang , Heung-Yeung Shum, Full-Frame Video Stabilization with Motion Inpainting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1150-1163, July 2006 [doi>10.1109/TPAMI.2006.141]
• 22: Meingast, M., Geyer, C., and Sastry, S. 2005. Geometric models of rolling-shutter cameras. In 6th Int. workshop on Omnidirectional vision, Camera networks, and non-classical cameras.
• 23: Morimoto, C., and Chellappa, R. 1997. Evaluation of image stabilization algorithms. In DARPA Image Understanding Workshop DARPA97, 295--302.
• 24: Richard M. Murray , S. Shankar Sastry , Li Zexiang, A Mathematical Introduction to Robotic Manipulation, CRC Press, Inc., Boca Raton, FL, 1994
• 25: David Nistér, Preemptive RANSAC for Live Structure and Motion Estimation, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.199, October 13-16, 2003
• 26: Michael Rubinstein , Ariel Shamir , Shai Avidan, Improved seam carving for video retargeting, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 27: Scott Schaefer , Travis McPhail , Joe Warren, Image deformation using moving least squares, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 28: Thorsten Thormählen , Hans-Peter Seidel, 3D-modeling by ortho-image generation from image sequences, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 29: Philip H. S. Torr , Andrew W. Fitzgibbon , Andrew Zisserman, The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences, International Journal of Computer Vision, v.32 n.1, p.27-44, Aug. 1. 1999 [doi>10.1023/A:1008140928553]
• 30: Lorenzo Torresani , Aaron Hertzmann , Chris Bregler, Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.5, p.878-892, May 2008 [doi>10.1109/TPAMI.2007.70752]
• 31: Anton van den Hengel , Anthony Dick , Thorsten Thormählen , Ben Ward , Philip H. S. Torr, VideoTrace: rapid interactive scene modelling from video, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 32: Yu-Shuen Wang , Chiew-Lan Tai , Olga Sorkine , Tong-Yee Lee, Optimized scale-and-stretch for image resizing, ACM Transactions on Graphics (TOG), v.27 n.5, December 2008
• 33: Wexler, Y., Shechtman, E., and Irani, M. 2004. Space-time video completion. In 2004 Conference on Computer Vision and Pattern Recognition (CVPR 2004), 120--127.
• 34: Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Nonhomogeneous content-driven video-retargeting. In IEEE International Conference on Computer Vision, 1--6.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Algorithms, Design, Measurement