Video retargeting: automating pan and scan

ABSTRACT

When a video is displayed on a smaller display than originally intended, some of the information in the video is necessarily lost. In this paper, we introduce Video Retargeting that adapts video to better suit the target display, minimizing the important information lost. We define a framework that measures the preservation of the source material, and methods for estimating the important information in the video. Video retargeting crops each frame and scales it to fit the target display. An optimization process minimizes information loss by balancing the loss of detail due to scaling with the loss of content and composition due to cropping. The cropping window can be moved during a shot to introduce virtual pans and cuts, subject to constraints that ensure cinematic plausibility. We demonstrate results of adapting a variety of source videos to small display sizes.

REFERENCES

• 1: D. Arijon. Grammar of the Film Language. Silman-James Press; Reprint edition, 1991.
• 2: William H. Bares , James C. Lester, Intelligent multi-shot visualization interfaces for dynamic 3D worlds, Proceedings of the 4th international conference on Intelligent user interfaces, p.119-126, January 05-08, 1999, Los Angeles, California, United States [doi>10.1145/291080.291101]
• 3: Michael Bianchi, Automatic video production of lectures using an intelligent and aware environment, Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia, p.117-123, October 27-29, 2004, College Park, Maryland [doi>10.1145/1052380.1052397]
• 4: J. Cantine, B. Lewis, and S. Howard. Shot by Shot; A Practical Guide to Filmmaking. Pittsburgh Filmmakers, 1995.
• 5: L.-Q. Chen, X. Xie, X. Fan, W.-Y. Ma, H.-J. Zhang, and H.-Q. Zhou. A visual attention model for adapting images on small displays. ACM Multimedia Systems Journal, 9(4):353--364, November 2003.
• 6: Franklin C. Crow, Summed-area tables for texture mapping, Proceedings of the 11th annual conference on Computer graphics and interactive techniques, p.207-212, January 1984
• 7: Xin Fan , Xing Xie , He-Qin Zhou , Wei-Ying Ma, Looking into video frames on small displays, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957063]
• 8: Andreas Girgensohn , John Boreczky , Patrick Chiu , John Doherty , Jonathan Foote , Gene Golovchinsky , Shingo Uchihashi , Lynn Wilcox, A semi-automatic approach to home video editing, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.81-89, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354415]
• 9: Li-wei He , Michael F. Cohen , David H. Salesin, The virtual cinematographer: a paradigm for automatic real-time camera control and directing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.217-224, August 1996 [doi>10.1145/237170.237259]
• 10: R. Heck, M. Wallick, and M. Gleicher. Virtual videography. ACM Transactions on Multimedia Computing, Communications and Applications, 3(1), 2007. to appear.
• 11: Yiqun Hu , Deepu Rajan , Liang-Tien Chia, Robust subspace analysis for detecting visual attention regions in images, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101306]
• 12: L. Itti and C. Koch. Computational modeling of visual attention. Nature Reviews Neuroscience, 2(3):194--203, March 2001.
• 13: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 14: S. Katz. Film Directing: Shot by Shot: Visualizing from Concept to Screen. Michael Wiese Productions, 1991.
• 15: Feng Liu , Michael Gleicher, Automatic image retargeting with fisheye-view warping, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095061]
• 16: B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proc. of International Joint Conference on Artificial Intelligence, pages 674--679, 1981.
• 17: Y.-F. Ma and H.-J. Zhang. A model of motion attention for video skimming. In Proc. IEEE ICIP, pages 129--132, 2002.
• 18: Yu-Fei Ma , Hong-Jiang Zhang, Contrast-based image attention analysis by using fuzzy growing, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957094]
• 19: H. Nothdurft. Salience from feature contrast: additivity across dimensions. Vision Research, 40(11-12):1183--1201, 2000.
• 20: H. Nothdurft. Salience from feature contrast: variations with texture density. Vision Research, 40(23):3181--3200, 2000.
• 21: W. R. and H. T. Fast camera motion analysis in mpeg domain. In Proc. of IEEE ICIP, pages 691--694, 1999.
• 22: M. Rasheed, Z. and Shah. Scene detection in hollywood movies and tv shows. In Proc. of IEEE CVPR, pages 343--348, 2003.
• 23: R. Rosenholtz. A simple saliency model predicts a number of motion popout phenomena. Vision Research, 39(19):3157--3163, 1999.
• 24: Yong Rui , Liwei He , Anoop Gupta , Qiong Liu, Building an intelligent camera management system, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500145]
• 25: Anthony Santella , Maneesh Agrawala , Doug DeCarlo , David Salesin , Michael Cohen, Gaze-based interaction for semi-automatic photo cropping, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124886]
• 26: Vidya Setlur , Saeko Takagi , Ramesh Raskar , Michael Gleicher , Bruce Gooch, Automatic image retargeting, Proceedings of the 4th international conference on Mobile and ubiquitous multimedia, December 08-10, 2005, Christchurch, New Zealand [doi>10.1145/1149488.1149499]
• 27: Bongwon Suh , Haibin Ling , Benjamin B. Bederson , David W. Jacobs, Automatic thumbnail cropping and its effectiveness, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.95-104, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964707]
• 28: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 29: P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. of IEEE CVPR, pages 511--518, 2001.
• 30: J. Wang, M. Reinders, R. Lagendijk, J. Lindenberg, and M. Kankanhalli. Video content presentation on tiny devices. In Proc. of IEEE Conference on Multimedia and Expo, pages 1711--1714, 2004.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

General Terms:
Algorithms, Human Factors

Keywords:
importance estimation, mobile multimedia, video editing, video retargeting