Re-cinematography: Improving the camerawork of casual video

ABSTRACT

This article presents an approach to postprocessing casually captured videos to improve apparent camera movement. Re-cinematography transforms each frame of a video such that the video better follows cinematic conventions. The approach breaks a video into shorter segments. Segments of the source video where there is no intentional camera movement are made to appear as if the camera is completely static. For segments with camera motions, camera paths are keyframed automatically and interpolated with matrix logarithms to give velocity-profiled movements that appear intentional and directed. Closeups are inserted to provide compositional variety in otherwise uniform segments. The approach automatically balances the tradeoff between motion smoothness and distortion to the original imagery. Results from our prototype show improvements to poor quality home videos.

REFERENCES

• 1: Radhakrishna S. V. Achanta , Wei-Qi Yan , Mohan S. Kankanhalli, Modeling Intent for Home Video Repurposing, IEEE MultiMedia, v.13 n.1, p.46-55, January 2006 [doi>10.1109/MMUL.2006.12]
• 2: Brett Adams , Svetha Venkatesh , Ramesh Jain, IMCE: Integrated media creation environment, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.1 n.3, p.211-247, August 2005 [doi>10.1145/1083314.1083315]
• 3: Marc Alexa, Linear combination of transformations, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 4: Tom Ang, Digital Video Handbook, Dorling Kindersley Publishing, Incorporated, 2005
• 5: Arijon, D. 1991. Grammar of the Film Language. Silman-James Press.
• 6: Eric P. Bennett , Leonard McMillan, Proscenium: a framework for spatio-temporal video editing, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957049]
• 7: Alessandro Bevilacqua , Pietro Azzari, High-Quality Real Time Motion Detection Using PTZ Cameras, Proceedings of the IEEE International Conference on Video and Signal Based Surveillance, p.23, November 22-24, 2006 [doi>10.1109/AVSS.2006.57]
• 8: Block, B. A. 2001. The Visual Story: Seeing the Structure of Film, TV, and New Media. Focal Press.
• 9: Bordwell, D. and Thompson, K. 1997. Film Art: An Introduction. McGraw-Hill.
• 10: Bob Brandon, The Complete Digital Video Guide: A Step-by-Step Handbook for Making Great Home Movies Using Your Digital Camcorder, Readers Digest, 2005
• 11: Brown, B. 2002. Cinematography: Theory and Practice: Imagemaking for Cinematographers, Directors & Videographers. Butterworth-Heinemann.
• 12: Buehler, C., Bosse, M., and McMillan, L. 2001. Non-metric image-based rendering for video stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Vol. 2. 609--614.
• 13: Juan Casares , A. Chris Long , Brad A. Myers , Rishi Bhatnagar , Scott M. Stevens , Laura Dabbish , Dan Yocum , Albert Corbett, Simplifying video editing using metadata, Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques, June 25-28, 2002, London, England [doi>10.1145/778712.778737]
• 14: Chalfen, R. 1987. Snapshot Versions of Life. Bowling Green State University Press.
• 15: Chen, L.-Q., Xie, X., Fan, X., Ma, W.-Y., Zhang, H.-J., and Zhou, H.-Q. 2003. A visual attention model for adapting images on small displays. Multimed. Syst. 9, 4, 353--364.
• 16: Christie, M., Machap, R., Normand, J.-M., Olivier, P., and Pickering, J. 2005. Virtual camera planning: A survey. In Proceedings of Smart Graphics. 40--52.
• 17: Franklin C. Crow, Summed-area tables for texture mapping, Proceedings of the 11th annual conference on Computer graphics and interactive techniques, p.207-212, January 1984
• 18: Dony, R., Mateer, J., and Robinson, J. 2005. Techniques for automated reverse storyboarding. IEE J. Vision, Image Signal Process. 152, 4, 425--436.
• 19: Martin A. Fischler , Robert C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, v.24 n.6, p.381-395, June 1981 [doi>10.1145/358669.358692]
• 20: Andreas Girgensohn , John Boreczky , Patrick Chiu , John Doherty , Jonathan Foote , Gene Golovchinsky , Shingo Uchihashi , Lynn Wilcox, A semi-automatic approach to home video editing, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.81-89, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354415]
• 21: M. Gleicher, Projective registration with difference decomposition, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.331, June 17-19, 1997
• 22: Michael L. Gleicher , Feng Liu, Re-cinematography: improving the camera dynamics of casual video, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291246]
• 23: Govindu, V. M. 2004. Lie-algebraic averaging for globally consistent motion estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
• 24: Hansen, M. and Mcdowel, L. 2001. Apparatus and method for removing blank areas from real-time stabilized images by inserting background information. U.S. Patent 6211913.
• 25: Rachel Heck , Michael Wallick , Michael Gleicher, Virtual videography, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.1, p.4-es, February 2007 [doi>10.1145/1198302.1198306]
• 26: Hua, X.-S., Lu, L., and Zhang, H.-J. 2004. Optimization-based automated home video editing system. IEEE Trans. Circ. Syst. Video Tech. 14, 5 (May), 572--583.
• 27: Irani, M. and Anandan, P. 1998. Video indexing based on mosaic representations. Proc. IEEE 86, 5, 905--921.
• 28: M. Irani , P. Anandan , S. Hsu, Mosaic based representations of video sequences and their applications, Proceedings of the Fifth International Conference on Computer Vision, p.605, June 20-23, 1995
• 29: Irani, M., Rousso, B., and Peleg, S. 1994. Recovery of ego-motion using image stabilization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 454--460.
• 30: Johnston, O. and Thomas, F. 1981. The Illusion of Life: Disney Animation. Abbeville Press.
• 31: Katz, S. D. 1991. Film Directing Shot by Shot: Visualizing from Concept to Screen. Michael Wiese Productions.
• 32: Kavan, L., Collins, S., O'Sullivan, C., and Zara, J. 2006. Dual quaternions for rigid transformation blending. Tech. Rep. TCD-CS-2006-46, Trinity College Dublin.
• 33: Kender, J. and Yeo, B.-L. 2000. On the structure and analysis of home video. In Proceedings of ACCV.
• 34: David Kirk , Abigail Sellen , Richard Harper , Ken Wood, Understanding videowork, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240634]
• 35: Litvin, A., Konrad, J., and Karl, W. 2003. Probabilistic video stabilization using kalman filtering and mosaicking. In Proceedings of the IS&T/SPIE Symposium on Electronic Imaging, Image, and Video Comm. 663--674.
• 36: Feng Liu , Michael Gleicher, Video retargeting: automating pan and scan, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180702]
• 37: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 38: Yasuyuki Matsushita , Eyal Ofek , Weina Ge , Xiaoou Tang , Heung-Yeung Shum, Full-Frame Video Stabilization with Motion Inpainting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.7, p.1150-1163, July 2006 [doi>10.1109/TPAMI.2006.141]
• 39: Tao Mei , Xian-Sheng Hua , He-Qin Zhou, Tracking users' capture intention: a novel complementary view for home video content analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101269]
• 40: Krystian Mikolajczyk , Cordelia Schmid, A Performance Evaluation of Local Descriptors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.10, p.1615-1630, October 2005 [doi>10.1109/TPAMI.2005.188]
• 41: David Nistér, Preemptive RANSAC for live structure and motion estimation, Machine Vision and Applications, v.16 n.5, p.321-329, December 2005 [doi>10.1007/s00138-005-0006-y]
• 42: Nocedal, J. and Wright, S. J. 2006. Numerical Optimization, 2nd ed. Springer.
• 43: Mihai Osian , Luc Van Gool, Video shot characterization, Machine Vision and Applications, v.15 n.3, p.172-177, July 2004 [doi>10.1007/s00138-004-0141-x]
• 44: Zailiang Pan , Chong-Wah Ngo, Structuring home video by snippet detection and pattern parsing, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026723]
• 45: Rosenholtz, R. 1999. A simple saliency model predicts a number of motion popout phenomena. Vision Research 39, 19, 3157--3163.
• 46: Bongwon Suh , Haibin Ling , Benjamin B. Bederson , David W. Jacobs, Automatic thumbnail cropping and its effectiveness, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.95-104, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964707]
• 47: Richard Szeliski, Video Mosaics for Virtual Environments, IEEE Computer Graphics and Applications, v.16 n.2, p.22-30, March 1996 [doi>10.1109/38.486677]
• 48: Szeliski, R. 2006. Image alignment and stitching: A tutorial. Tech. rep. MSR-TR-2004-92, Microsoft Research.
• 49: Laura Teodosio , Walter Bender, Salient stills, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.1 n.1, p.16-36, February 2005 [doi>10.1145/1047936.1047940]
• 50: Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 511--518.
• 51: Yonatan Wexler , Eli Shechtman , Michal Irani, Space-Time Completion of Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.463-476, March 2007 [doi>10.1109/TPAMI.2007.60]
• 52: Daniel N. Wood , Adam Finkelstein , John F. Hughes , Craig E. Thayer , David H. Salesin, Multiperspective panoramas for cel animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.243-250, August 1997 [doi>10.1145/258734.258859]
• 53: Wei-Qi Yan , Mohan S Kankanhalli, Detection and removal of lighting & shaking artifacts in home videos, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641031]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Design, Experimentation, Human Factors

Keywords:
Image stabilization, casual video, cinematography