Rushes video summarization by object and event understanding

ABSTRACT

This paper explores a variety of visual and audio analysis techniques in selecting the most representative video clips for rushes summarization at TRECVID 2007. These techniques include object detection, camera motion estimation, keypoint matching and tracking, audio classification and speech recognition. Our system is composed of two major steps. First, based on video structuring, we filter undesirable shots and minimize theinter-shot redundancy by repetitive shot detection. Second, a representability measure is proposed to model the presence of objects and four audio-visual events: motion activity of objects, camera motion, scene changes,and speech content, in a video clip. The video clips with the highest representability scores are selected for summarization. The evaluation at TRECVID shows that our experimental results are highly encouraging, where we rank first in EA (easy to understand), second in RE (little redundancy) and third in IN (inclusion of objects and events).

REFERENCES

• 1: A. M. Ferman and A. M. Tekalp, "Two-stage hierarchical video summary extraction to match low-level user browsing preerences", IEEE Trans. on Multimedia, vol. 5, no. 2, pp. 244--256, 2003.
• 2: C. Gianluigi and S. Raimondo, "An Innovative Algorithm for Keyframe Extraction in Video Summarization", Journal of Real-Time Image Processing, vol. 1, no. 1, pp. 69--88, 2006.
• 3: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 4: C. W. Ngo, T. C. Pong, and R. T. Chin, "Video Partitioning through Temporal Slices Analysis", IEEE Trans. on Circuits and Systems for Video Technology, 11(8), pp. 941--953, 2001.
• 5: C. W. Ngo, Z. Pan, X. Wei, X. Wu, H. K. Tan, and W.Zhao, "Motion Driven Approaches to Shot Boundary Detection, Low-Level Feature Extraction and BBC Rushes Characterization at TRECVID 2005", TRECVID Workshop, 2005.
• 6: C. W. Ngo, Z. Pan, and X. Y.Wei, "Hierarchical Hidden Markov Model for Rushes Structuring and Indexing ", Int. Conf. on Image and Video Retrieval, 2006.
• 7: Chong-Wah Ngo , Wan-Lei Zhao , Yu-Gang Jiang, Fast tracking of near-duplicate keyframes in broadcast domain with transitivity propagation, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180827]
• 8: Z. Pan and C. W. Ngo, "Moving Object Detection, Association and Selection in Home Videos", IEEE Trans. on Multimedia, vol. 9, no. 2, Feb 2007.
• 9: Savitha Srinivasan , Dragutin Petkovic , Dulce Ponceleon, Towards robust features for classifying audio in the CueVideo system, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.393-400, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319658]
• 10: S. Tang, Y. Zhang, J. Li, X. Pan, T. Xia, and M. Li, "Rushes Exploitation 2006 By CAS MCG", TRECVID Workshop, 2006.
• 11: C. M. Taskiran, Z. Pizlo, A. Amir, D. Ponceleon, and E. Delp, "Automated Video Program Summarization Using Speech Transcripts", IEEE Trans. on Multimedia, vol. 8, no.4, pp.775--791, 2006.
• 12: W. L. Zhao, C. W. Ngo, H. K. Tan, and X. Wu, "Near-Duplicate Keyframe Identification with Interest Point Matching and Pattern Learning ", IEEE Trans. on Multimedia, to appear.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

Additional Classification:
  J. Computer Applications
  J.m MISCELLANEOUS

General Terms:
Algorithms, Design, Performance

Keywords:
event understanding, object detection, rushes video summarization