Re-cinematography: improving the camera dynamics of casual video

ABSTRACT

This paper presents an approach to post-processing casually captured videos to improve apparent camera movement. Re-cinematography transforms each frame of a video such that the video better follows cinematic conventions. The approach breaks videos into shorter segments. For segments of the source video where the camera is relatively static, re-cinematography uses image stabilization to make the result look locked-down. For segments with camera motions, camera paths are keyframed automatically and interpolated with matrix logarithms to give velocity-profiled movements that appear intentional and directed. The approach automatically balances the tradeoff between motion smoothness and distortion to the original imagery. Results from our prototype show improvements to poor quality home videos.

REFERENCES

• 1: Marc Alexa, Linear combination of transformations, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 2: D. Arijon. Grammar of the Film Language. Silman-James Press, 1991.
• 3: Eric P. Bennett , Leonard McMillan, Proscenium: a framework for spatio-temporal video editing, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957049]
• 4: Alessandro Bevilacqua , Pietro Azzari, High-Quality Real Time Motion Detection Using PTZ Cameras, Proceedings of the IEEE International Conference on Video and Signal Based Surveillance, p.23, November 22-24, 2006 [doi>10.1109/AVSS.2006.57]
• 5: B. A. Block. The Visual Story: Seeing the Structure of Film, Tv, and New Media. Focal Press, 2001.
• 6: D. Bordwell and K. Thompson. Film Art: An Introduction. The McGraw-Hill Companies, Inc., 1997.
• 7: B. Brown. Cinematography: Theory and Practice: Imagemaking for Cinematographers, Directors & Videographers. Butterworth-Heinemann, 2002.
• 8: C. Buehler, M. Bosse, and L. McMillan. Non-metric image-based rendering for video stabilization. In IEEE CVPR, volume 2, pages 609--614, 2001.
• 9: L.-Q. Chen, X. Xie, X. Fan, W.-Y. Ma, H.-J. Zhang, and H.-Q. Zhou. A visual attention model for adapting images on small displays. ACM Multimedia Systems Journal, pages 353--364, 2003.
• 10: M. Christie, R. Machap, J.-M. Normand, P. Olivier, and J. Pickering. Virtual camera planning: A survey. In Proceedings Smart Graphics, pages 40--52, 2005.
• 11: Deshaker. www.guthspot.se/video/deshaker.htm, 2006.
• 12: R. Dony, J. Mateer, and J. Robinson. Techniques for automated reverse storyboarding. IEE Journal of Vision, Image and Signal Processing, 152(4):425--436, 2005.
• 13: Andreas Girgensohn , John Boreczky , Patrick Chiu , John Doherty , Jonathan Foote , Gene Golovchinsky , Shingo Uchihashi , Lynn Wilcox, A semi-automatic approach to home video editing, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.81-89, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354415]
• 14: M. Gleicher, Projective registration with difference decomposition, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.331, June 17-19, 1997
• 15: V. M. Govindu. Lie-algebraic averaging for globally consistent motion estimation. In IEEE CVPR, 2004.
• 16: Rachel Heck , Michael Wallick , Michael Gleicher, Virtual videography, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.1, p.4-es, February 2007 [doi>10.1145/1198302.1198306]
• 17: X.-S. Hua, L. Lu, and H.-J. Zhang. Optimization-based automated home video editing system. IEEE Transactions on Circuits and Systems for Video Technology, 14(5):572--583, May 2004.
• 18: M. Irani and P. Anandan. Video indexing based on mosaic representations. Proc. IEEE, 86(5):905--921, 1998.
• 19: M. Irani , P. Anandan , S. Hsu, Mosaic based representations of video sequences and their applications, Proceedings of the Fifth International Conference on Computer Vision, p.605, June 20-23, 1995
• 20: M. Irani, B. Rousso, and S. Peleg. Recovery of ego-motion using image stabilization. In IEEE CVPR, pages 454--460, 1994.
• 21: O. Johnston and F. Thomas. The Illusion of Life: Disney Animation. Abbeville Press, 1981.
• 22: S. D. Katz. Film directing shot by shot: visualizing from concept to screen. Michael Wiese Productions, 1991.
• 23: L. Kavan, S. Collins, C. O'Sullivan, and J. Zara. Dual quaternions for rigid transformation blending. Technical Report TCD-CS-2006-46, Trinity College Dublin, 2006.
• 24: A. Litvin, J. Konrad, and W. Karl. Probabilistic video stabilization using kalman filtering and mosaicking. In Proc. IS&T/SPIE Symp. Electronic Imaging, Image, and Video Comm., pages 663--674, 2003.
• 25: Feng Liu , Michael Gleicher, Video retargeting: automating pan and scan, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180702]
• 26: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 27: Yasuyuki Matsushita , Eyal Ofek , Weina Ge , Xiaoou Tang , Heung-Yeung Shum, Full-Frame Video Stabilization with Motion Inpainting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1150-1163, July 2006 [doi>10.1109/TPAMI.2006.141]
• 28: David Nistér, Preemptive RANSAC for live structure and motion estimation, Machine Vision and Applications, v.16 n.5, p.321-329, December 2005 [doi>10.1007/s00138-005-0006-y]
• 29: R. Rosenholtz. A simple saliency model predicts a number of motion popout phenomena. Vision Research, 39(19):3157--3163, 1999.
• 30: Bongwon Suh , Haibin Ling , Benjamin B. Bederson , David W. Jacobs, Automatic thumbnail cropping and its effectiveness, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.95-104, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964707]
• 31: R. Szeliski. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22--30, 1996.
• 32: R. Szeliski. Image alignment and stitching: A tutorial. Technical Report MSR-TR-2004-92, Microsoft Research, 2006.
• 33: Laura Teodosio , Walter Bender, Salient stills, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.1 n.1, p.16-36, February 2005 [doi>10.1145/1047936.1047940]
• 34: P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In IEEE CVPR, pages 511--518, 2001.
• 35: Yonatan Wexler , Eli Shechtman , Michal Irani, Space-Time Completion of Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.463-476, March 2007 [doi>10.1109/TPAMI.2007.60]
• 36: Daniel N. Wood , Adam Finkelstein , John F. Hughes , Craig E. Thayer , David H. Salesin, Multiperspective panoramas for cel animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.243-250, August 1997 [doi>10.1145/258734.258859]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Experimentation, Human Factors

Keywords:
casual video, cinematography, image stabilization