Video Manga: generating semantically meaningful video summaries

ABSTRACT

This paper presents methods for automatically creating pictorial video summaries that resemble comic books. The relative importance of video segments is computed from their length and novelty. Image and audio analysis is used to automatically detect and emphasize meaningful events. Based on this importance measure, we choose relevant keyframes. Selected keyframes are sized by importance, and then efficiently packed into a pictorial summary. We present a quantitative measure of how well a summary captures the salient events in a video, and show how it can be used to improve our summaries. The result is a compact and visually pleasing summary that captures semantically important events, and is suitable for printing or Web access. Such a summary can be further enhanced by including text captions derived from OCR or other methods. We describe how the automatically generated summaries are used to simplify access to a large collection of videos.

REFERENCES

• 1: Philippe Aigrain , Philippe Joly , Véronique Longueville, Medium knowledge-based macro-segmentation of video into sequences, Intelligent multimedia information retrieval, MIT Press, Cambridge, MA, 1997
• 2: F. Arman , R. Depommier , A. Hsu , M.-Y. Chiu, Content-based browsing of video sequences, Proceedings of the second ACM international conference on Multimedia, p.97-103, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192630]
• 3: Boreczky, J. and Rowe, L., "Comparison of Video Shot Boundary Detection Techniques," in Proc. SPIE Conference on Storage and Retrieval for Still Image and 'Video Databases IE, San Jose, CA, February, 1996, pp. t70-179.
• 4: Michael G. Christel , Michael A. Smith , C. Roy Taylor , David B. Winkler, Evolving video skims into useful multimedia abstractions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.171-178, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274670]
• 5: Jonathan Foote , John Boreczhy , Andreas Girgensohn , Lynn Wilcox, An intelligent media browser using automatic multimodal analysis, Proceedings of the sixth ACM international conference on Multimedia, p.375-380, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290804]
• 6: Foote, J., Boreczky, J., and Wilcox, L., "Finding Presentations in Recorded Meetings Using Audio and Video Features," in Proc. 1CASSP '99, Vol. 6, pp. 3045-3048, 1999.
• 7: Girgensohn, A. and Boreczky, J., "Time-Constrained Keyframe Selection Technique," in IEEE Multimedia Systems '99, IEEE Computer Society, Vol. 1, pp. 756- 761, 1999.
• 8: Girgensohn, A. and Foote, J., "Video Frame Classification Using Transform Coefficients," in Proc. ICASSP '99, Vol. 6, pp. 3045-3048, 1999.
• 9: Huang, Q., Liu, Z. and Rosenberg, A., "Automated Semantic Structure Reconstruction and Representation Generation for Broadcast News," in Proc. iS& T/SPIE Conference on Storage and Retrieval for Image and Video Databases VII, Vol. 3656, pp. 50-62, 1999.
• 10: ISO MPEG 7 Content Set, Item V20, "Korea's Pop Singers' Live Music Show", Korean Broadcasting System, 1998.
• 11: Pfeiffer, S., Lienhart, R., Fischer, S. and Effelsberg, W., "Abstracting digital movies automatically," in Journal of Visual Communication and Image Representation, 7(4), pp. 345-353, December 1996.
• 12: Edie Rasmussen, Clustering algorithms, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 13: Behzad Shahraray , David C. Gibbon, Automated authoring of hypermedia documents of video programs, Proceedings of the third ACM international conference on Multimedia, p.401-409, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215304]
• 14: Michael A. Smith, Video Skimming and Characterization through the Combination of Image and Language Understanding Techniques, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.775, June 17-19, 1997
• 15: Yukinobu Taniguchi , Akihito Akutsu , Yoshinobu Tonomura, PanoramaExcerpts: extracting and packing panoramas for video browsing, Proceedings of the fifth ACM international conference on Multimedia, p.427-436, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266396]
• 16: Uchihashi, S. and Foote, J., "Summarizing Video Using a Shot Importance Measure and a Frame-Packing Algorithm," in Proc. ICASSP '99, Vol. 6, pp. 3041- 3044, 1999.
• 17: Yeo, B-L. and Yeung, M., "Classification, Simplification and Dynamic Visualization of Scene Transition Graphs for Video Browsing," in Proc. IS&T/SPIE Electronic Imaging 98: Storage and Retrieval for Image and Wdeo Databases VI.
• 18: Yeung, M. M., Yeo, B. L., Wolf, W. and Liu, B., "Video Browsing using Clustering and Scene Transitions on Compressed Sequences," in SPIE Vol. 2417 Multimedia Computing and Networking 1995, pp. 399-413, Feb. 1995.
• 19: Yeung, M. and Yeo, B-L., "Video Visualization for Compact Presentation and Fast Browsing of Pictorial Content," in IEEE Trans. Circuits and Svs. for Video Technology, Vol. 7, No. 5, pp. 771-785, Oct. 1997.
• 20: Yu, H., Clark, C., Malkin, R. and Waibel, A., "Experiments In Automatic Meeting Transcription Using JRTK," in Proc. ICASSP 98, pp. 921-924, 1998.
• 21: H. J. Zhang , C. Y. Low , S. W. Smoliar , J. H. Wu, Video parsing, retrieval and browsing: an integrated and content-based solution, Proceedings of the third ACM international conference on Multimedia, p.15-24, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215068]
• 22: Zhuang, Y., Rui, Y., Huang, T.S. and Mehrotra, S., "Adaptive Key Frame Extraction Using Unsupervised Clustering," in Proc. ICIP '98, Vol. I, pp. 866-870, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Design, Experimentation, Theory

Keywords:
keyframe selection and layout, video summarization and analysis