The trecvid 2007 BBC rushes summarization evaluation pilot

ABSTRACT

This paper provides an overview of a pilot evaluation of video summaries using rushes from several BBC dramatic series. It was carried out under the auspices of TRECVID. Twenty-two research teams submitted video summaries of up to 4% duration, of 42 individual rushes video files aimed at compressing out redundant and insignificant material. The output of two baseline systems built on straightforward content reduction techniques was contributed by Carnegie Mellon University as a control. Procedures for developing ground truth lists of important segments from each video were developed at Dublin City University and applied to the BBC video. At NIST each summary was judged by three humans with respect to how much of the ground truth was included, how easy the summary was to understand, and how much repeated material the summary contained. Additional objective measures included: how long it took the system to create the summary, how long it took the assessor to judge it against the ground truth, and what the summary's duration was. Assessor agreement on finding desired segments averaged 78% and results indicate that while it is difficult to exceed the performance of baselines, a few systems did.

REFERENCES

• 1: Michael G. Christel , Michael A. Smith , C. Roy Taylor , David B. Winkler, Evolving video skims into useful multimedia abstractions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.171-178, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274670]
• 2: Ding, W., Marchionini, G., & Tse, T. (1997). Previewing Video Data: Browsing Key Frames at High Rates Using a Video Slide Show. In Proceedings of the International Symposium on Research, Development and Practice in Digital Libraries (pp. 151--158). Tsukuba Science City, Japan.
• 3: Ekin, A., Tekalp, A. M., & Mehrotra, R. (2003). Automatic Soccer Video Analysis and Summarization. IEEE Transactions on Image Processing, 12 (7), 796--807.
• 4: Ferman, A. M., & Tekalp, A. M. (2003). Two-Stage Hierarchical Video Summary Extraction to Match Low-Level User Browsing Preferences. IEEE Transactions on Multimedia, 5 (2), 244--256.
• 5: Hauptmann, A. (2007). Personal communication, School of Computer Science at Carnegie Mellon University.
• 6: Liwei He , Elizabeth Sanocki , Anoop Gupta , Jonathan Grudin, Auto-summarization of audio-video presentations, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.489-498, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319691]
• 7: Anita Komlodi , Gary Marchionini, Key frame preview techniques for video browsing, Proceedings of the third ACM conference on Digital libraries, p.118-125, June 23-26, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276675.276688]
• 8: Manly, B. F. J. (1997). Randomization, Bootstrap, and Monte Carlo Methods in Biology (2nd ed.). London, UK: Chapman & Hall.
• 9: Gary Marchionini, Human performance measures for video retrieval, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178720]
• 10: mplayer. (2007). Mplayer - the Movie Player. URL: www.mplayerhq.hu/design7/news.html.
• 11: Smeaton, A. F., & Over, P. (2003). TRECVID: Benchmarking the effectiveness of information retrieval tasks on digital video. Proc. of the International Conference on Image and Video Retrieval (CIVR).
• 12: Alan F. Smeaton , Paul Over , Wessel Kraaij, TRECVID: evaluating the effectiveness of information retrieval tasks on digital video, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027678]
• 13: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 14: Taskiran, C. M., Pizlo, Z., Amir, A., Ponceleon, D., & Delp, E. J. (2006). Automated video program summarization using speech transcripts. IEEE Transactions on Multimedia, 8 (4), 775--791.
• 15: Ba Tu Truong , Svetha Venkatesh, Video abstraction: A systematic review and classification, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.1, p.3-es, February 2007 [doi>10.1145/1198302.1198305]
• 16: Wright, R. (2005). Personal communication, Technology Manager, Projects, BBC Information & Archives.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Experimentation

Keywords:
benchmarking, evaluation, summarization, video