Near-lossless video summarization

ABSTRACT

The daunting yet increasing volume of videos on the Internet brings the challenges of storage and indexing to existing online video services. Current techniques like video compression and summarization are still struggling to achieve the two often conflicting goals of low storage and high visual and semantic fidelity. In this work, we develop a new system for video summarization, called "Near-Lossless Video Summarization" (NLVS), which is able to summarize a video stream with the least information loss by using an extremely small piece of metadata. The summary consists of a set of synthesized mosaics and representative keyframes, a compressed audio stream, as well as the metadata about video structure and motion. Although at a very low compression ratio (i.e., 1/30 of H.264 baseline in average, where traditional compression techniques like H.264 fail to preserve the fidelity), the summary still can be used to reconstruct the original video (with the same duration) nearly without semantic information loss. We show that NLVS is a powerful tool for significantly reducing video storage through both objective and subjective comparisons with state-of-the-art video compression and summarization techniques.

REFERENCES

• 1: TREC video retrieval evaluation. http://www--nlpir.nist.gov/projects/trecvid/.
• 2: MPEG-1 VideoGroup, Information technology--coding of moving pictures and associated audio for digital storage media up to about 1.5 Mbit/s: Part 2--Video. ISO/IEC 11172-2, 1993.
• 3: MPEG-2 Video Group, Information technology--generic coding of moving pictures and associated audio: Part 2--Video. ISO/IEC 13818-2, 1995.
• 4: MPEG-4 Video Group, Generic coding of audio-visual objects: Part 2--Visual. ISO/IEC JTC1/SC29/WG11 N1902, FDIS of ISO/IEC 14 496-2, 1998.
• 5: ITU-T Rec. H.264 | ISO/IEC 14496-10 AVC, Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification. March 2003.
• 6: ITU-T Rec. H.263, Video coding for low bit rate communication. version 1, Nov. 1995; version 2, Jan. 1998; version 3, Nov. 2000.
• 7: 3rd Generation Partnership Project. AMR speech codec: General description. TS 26.071 version 5.0.0, June 2002.
• 8: Bing. http://www.bing.com/.
• 9: P. Bouthemy, M. Gelgon, and F. Ganansia. A unified approach to shot change detection and camera motion characterization. IEEE Trans. on Circuit and Syst. for Video Tech., 9(7):1030--1044, 1999.
• 10: T. Bray, J. Paoli, C. M. Sperberg-McQueen, and E. Maler. Extensible Markup Language (XML) 1.0 (Second Edition). Available at http://www.w3.org/TR/REC-xml, 2000.
• 11: L. Carter. Web could collapse as video demand soars. Daily Telegraph, http://www.telegraph.co.uk/news/uknews/1584230/Web-could-collapse-as-video-demand-soars.html.
• 12: W. A. C.Fernando, C. N. Canagarajah, and D. R. Bull. Automatic detection of fade--in and fade-out in video sequences. Proceedings of IEEE International Symposium on Circuits and Systems, pages 255--258, 1999.
• 13: Digital Compression and Coding of Continuoustone Still Images, Part 1. Requirements and guidelines. ISO/IEC JTC1 Draft International Standard 10918-1, Nov. 1991.
• 14: A. G. Hauptmann, M. G. Christel, W.-H. Lin, and etc. Clever clustering vs. simple speed-up for summarizing rushes. Proceedings of the International Workshop on TRECVID Video Summarization, pages 20--24, 2007.
• 15: X.-S. Hua, L. Lu, and H.-J. Zhang. Optimization--based automated home video editing system. IEEE Trans. on Circuit and Syst. for Video Tech., 14(5):572--583, 2004.
• 16: Hulu. http://www.hulu.com/.
• 17: M. Irani and P. Anandan. Video indexing based on mosaic representations. Proceedings of the IEEE, 86(5):905--921, 1998.
• 18: JVT Reference Software version 15.1A. http://bs.hhi.de/suehring/tml/.
• 19: C. Kim and J.-N. Hwang. Object--based video abstraction for video surveillance systems. IEEE Trans. on Circuit and Syst. for Video Tech., 12(12):1128--1138, 2002.
• 20: J. G. Kim, H. S. Chang, J. Kim, and H. M. Kim. Efficient camera motion characterization for MPEG video indexing. In Proceedings of ICME, pages 1171--1174, 2000.
• 21: J. Konrad and F. Dufaux. Improved global motion estimation for N3. ISO/IEC JTC1/SC29/WG11 M3096, 1998.
• 22: Y.-F. Ma, L. Lu, H.-J. Zhang, and M. Li. A user attention model for video summarization. Proceedings of ACM Multimedia, 2002.
• 23: R. Marzotto, A. Fusiello, and V. Murino. High resolution video mosaicing with global alignment. Proceeding of CVPR, pages 692--698, 2004.
• 24: T. Mei, X.-S. Hua, W. Lai, L. Yang, and et al. MSRA-USTC-SJTU at TRECVID 2007: High-level feature extraction and search. In TREC Video Retrieval Evaluation Online Proceedings, 2007.
• 25: T. Mei, X.-S. Hua, H.-Q. Zhou, S. Li, and H.-J. Zhang. Efficient video mosaicing based on motion analysis. In Proceedings of IEEE International Conference on Image Processing, pages 861--864, 2005.
• 26: T. Mei, X.-S. Hua, C.-Z. Zhu, H.-Q. Zhou, and S. Li. Home video visual quality assessment with spatiotemporal factors. IEEE Trans. on Circuit and Syst. for Video Tech., 17(6):699--706, June 2007.
• 27: Metacafe. http://www.metacafe.com/.
• 28: Revver. http://www.revver.com/.
• 29: E. Schonfeld. YouTube's Chad Hurley: "We Have The Largest Library of HD Video On The Internet.". TechCrunch. http://www.techcrunch.com/2009/01/30/youtubes-chad-hurley-we-have-the-largest-library-of-hd-video-on-the-internet/.
• 30: B. T. Truong and S. Venkatesh. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl., 3(1):692--698, 2007.
• 31: T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the H.264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol., 13(7):560--576, July 2003.
• 32: YouTube. http://www.youtube.com/.
• 33: H.-J. Zhang. Content-Based Video Analysis, Retrieval and Browsing. Academic Press, 2002.
• 34: H.-J. Zhang, A. Kankanhalli, and S. W. Smoliar. Automatic partitioning of full-motion video. Multimedia Systems, 1(1):10--28, June 1993.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
online video service, video storage, video summarization