Feature fusion and redundancy pruning for rush video summarization

ABSTRACT

This paper presents a video summarization technique for rushes that employs high-level feature fusion to identify segments for inclusion. It aims to capture distinct video events using a variety of features: k-means based weighting, speech, camera motion, significant differences in HSV color space, and a dynamic time warping (DTW) based feature that suppresses repeated scenes. The feature functions are used to drive a weighted k-means based clustering to identify visually distinct, important segments that constitute the final summary. The optimal weights corresponding to the individual features are obtained using a gradient descent algorithm that maximizes the recall of ground truth events from representative training videos. Analysis reveals a lengthy computation time but high quality results (60% average recall over 42 test videos) as based on manually-judged inclusion ofdistinct shots. The summaries were judged relatively easy to view and had an average amount of redundancy.

REFERENCES

• 1: http://vision.ucla.edu/.vedaldi/code/bag/bag.html. UCLA Bag of Features.
• 2: P. Bouthemy, M. Gelgon, and F. Ganansia. A unified approach to shot change detection and camera motion characterization. IEEE Trans. on Circuits and Systems for Video Technology, 9(7):1030--1044, 1999.
• 3: Yihong Gong , Xin Liu, Video summarization and retrieval using singular value decomposition, Multimedia Systems, v.9 n.2, p.157-168, August 2003 [doi>10.1007/s00530-003-0086-3]
• 4: A. Hanjalic. Generic approach to highlights extraction from a sport video. In Proc. of ICIP (1), pages 1--4, 2003.
• 5: A. Hanjalic. Content-Based Analysis of Digital Video. Kluwer Academic Publishers, 2004.
• 6: Y. Li and C.-C. J. Kuo. Video Content Analysis using Multimodal Information. Kluwer Academic Publishers, 2003.
• 7: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 8: Lie Lu , Hao Jiang , HongJiang Zhang, A robust audio classification and segmentation method, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500173]
• 9: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 10: Yuichi Nakamura , Takeo Kanade, Semantic analysis for video contents extraction—spotting by association in news video, Proceedings of the fifth ACM international conference on Multimedia, p.393-401, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266391]
• 11: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 12: Paul Over , Alan F. Smeaton , Philip Kelly, The trecvid 2007 BBC rushes summarization evaluation pilot, Proceedings of the international workshop on TRECVID video summarization, p.1-15, September 28-28, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290031.1290032]
• 13: Lawrence Rabiner , Biing-Hwang Juang, Fundamentals of speech recognition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Experimentation, Performance

Keywords:
SIFT-based object detection, dynamic time warping, feature fusion, video summarization