The trecvid 2008 BBC rushes summarization evaluation

ABSTRACT

This paper describes an evaluation of automatic video summarization systems run on rushes from several BBC dramatic series. It was carried out under the auspices of the TREC Video Retrieval Evaluation (TRECVid) as a followup to the 2007 video summarization workshop held at ACM Multimedia 2007. 31 research teams submitted video summaries of 40 individual rushes video files, aiming to compress out redundant and insignificant material. Each summary had a duration of at most 2% of the original. The output of a baseline system, which simply presented each full video at 50 times normal speed was contributed by Carnegie Mellon University (CMU) as a control. The 2007 procedures for developing ground truth lists of important segments from each video were applied at the National Institute of Standards and Technology (NIST) to the BBC videos. At Dublin City University (DCU) each summary was judged by 3 humans with respect to how much of the ground truth was included and how well-formed the summary was. Additional objective measures included: how long it took the system to create the summary, how long it took the assessor to judge it against the ground truth, and what the summary's duration was. Assessor agreement on finding desired segments averaged 81%. Results indicated that while it was still difficult to exceed the performance of the baseline on including ground truth, the baseline was outperformed by most other systems with respect to avoiding redundancy/junk and presenting the summary with a pleasant tempo/rhythm.

REFERENCES

• 1: Werner Bailer , Georg Thallinger, Comparison of content selection methods for skimming rushes video, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.85-89, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463578]
• 2: Vítězslav Beran , Michal Hradis , Pavel Zemcik , Adam Herout , Ivo Řezníček, Video summarization at Brno university of technology, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.31-34, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463567]
• 3: Hervé Bredin , Daragh Byrne , Hyowon Lee , Noel E. O'Connor , Gareth J.F. Jones, Dublin City University at the TRECVid 2008 BBC rushes summarisation task, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.45-49, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463570]
• 4: Vasileios Chasanis , Aristidis Likas , Nikolaos Galatsanos, Video rushes summarization using spectral clustering and sequence alignment, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.75-79, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463576]
• 5: Francine Chen , John Adcock , Matthew Cooper, A simplified approach to rushes summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.60-64, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463573]
• 6: Michael G. Christel , Alexander G. Hauptmann , Wei-Hao Lin , Ming-Yu Chen , Jun Yang , Bryan Maher , Robert V. Baron, Exploring the utility of fast-forward surrogates for bbc rushes, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.35-39, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463568]
• 7: Marcin Detyniecki , Christophe Marsala, Adaptive acceleration and shot stacking for video rushes summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.109-113, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463583]
• 8: Emilie Dumont , Bernard Mérialdo, Sequence alignment for redundancy removal in video rushes summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.55-59, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463572]
• 9: Emilie Dumont , Bernard Mérialdo , Slim Essid , Werner Bailer , Herwig Rehatschek , Daragh Byrne , Hervé Bredin , Noel E. O'Connor , Gareth J.F Jones , Alan F. Smeaton , Martin Haller , Andreas Krutz , Thomas Sikora , Tomas Piatrik, Rushes video summarization using a collaborative approach, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.90-94, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463579]
• 10: Mehdi Ellouze , Hichem Karray , Adel M. Alimi, Regim, research group on intelligent machines, tunisia, at TRECVID 2008, BBC rushes summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.105-108, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463582]
• 11: David Gorisse , Frederic Precioso , Sylvie Philipp-Foliguet , Matthieu Cord, Summarization scheme based on near-duplicate analysis, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.50-54, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463571]
• 12: Robert Laganière , Raphael Bacco , Arnaud Hocevar , Patrick Lambert , Grégory Païs , Bogdan E. Ionescu, Video summarization from spatio-temporal features, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.144-148, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463590]
• 13: Narongsak Putpuek , Duy-Dinh Le , Nagul Cooharojananone , Shin'ichi Satoh , Chidchanok Lursinsap, Rushes summarization using different redundancy elimination approaches, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.100-104, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463581]
• 14: Yang Liu , Yan Liu , Tongwei Ren , Keith C.C. Chan, Rushes video summarization using audio-visual information and sequence alignment, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.114-118, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463584]
• 15: Zhu Liu , Eric Zavesky , Behzad Shahraray , David Gibbon , Andrea Basso, Brief and high-interest video summary generation: evaluating the AT&T labs rushes summarizations, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.21-25, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463565]
• 16: B. F. J. Manly. Randomization, Bootstrap, and Monte Carlo Methods in Biology. Chapman & Hall, London, UK, 2nd edition, 1997.
• 17: mplayer. Mplayer - the Movie Player. URL: www.mplayerhq.hu/design7/news.html, 2007.
• 18: S. U. Naci , Uros Damnjanovic , Boris Mansencal , Jenny Benois-Pineau , Christian Kaes , Marzia Corvaglia , Eliana Rossi , Naiara Aginako, The COST292 experimental framework for rushes summarization task in TRECVID 2008, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.40-44, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463569]
• 19: Akitsugu Noguchi , Keiji Yanai, Rushes summarization based on color, motion and face, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.139-143, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463589]
• 20: Paul Over , Alan F. Smeaton , Philip Kelly, The trecvid 2007 BBC rushes summarization evaluation pilot, Proceedings of the international workshop on TRECVID video summarization, p.1-15, September 28-28, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290031.1290032]
• 21: Georges Quénot , Jenny Benois-Pineau , Boris Mansencal , Eliana Rossi , Matthieu Cord , Frederic Precioso , David Gorisse , Patrick Lambert , Bertrand Augereau , Lionel Granjon , Denis Pellerin , Michèle Rombaut , Stephane Ayache, Rushes summarization by IRIM consortium: redundancy removal and multi-feature fusion, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.80-84, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463577]
• 22: Jinchang Ren , Jianmin Jiang , Christian Eckes, Hierarchical modeling and adaptive clustering for real-time summarization of rush videos in trecvid'08, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.26-30, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463566]
• 23: Reede Ren , P. Punitha , Joemon Jose, Video redundancy detection in rushes collection, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.65-69, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463574]
• 24: Masanori Sano , Yoshihiko Kawai , Nobuyuki Yagi , Shin'ichi Satoh, Video rushes summarization utilizing retake characteristics, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.95-99, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463580]
• 25: Johannes Sasongko , Cyril Rohr , Dian Tjondronegoro, Efficient generation of pleasant video summaries, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.119-123, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463585]
• 26: A. F. Smeaton and P. Over. TRECVID: Benchmarking the effectiveness of information retrieval tasks on digital video. In Proc. of the International Conference on Image and Video Retrieval (CIVR), pages 451--456. Springer, 2003.
• 27: Alan F. Smeaton , Paul Over , Wessel Kraaij, TRECVID: evaluating the effectiveness of information retrieval tasks on digital video, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027678]
• 28: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 29: C. M. Taskiran, Z. Pizlo, A. Amir, D. Ponceleon, and E. J. Delp. Automated video program summarization using speech transcripts. IEEE Transactions on Multimedia, 8(4):775---791, August 2006.
• 30: Pablo Toharia , Oscar David Robles , Luis Pastor , Angel Rodriguez, Combining activity and temporal coherence with low-level information for summarization of video rushes, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.70-74, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463575]
• 31: Ba Tu Truong , Svetha Venkatesh, Video abstraction: A systematic review and classification, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.1, p.3-es, February 2007 [doi>10.1145/1198302.1198305]
• 32: Víctor Valdés , José M. Martínez, Binary tree based on-line video summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.134-138, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463588]
• 33: Tao Wang , Shangping Feng , Patricia P. Wang , Wei Hu , Shuang Zhang , Wei Zhang , Yangzhou Du , Jianguo Li , Jianmin Li , Yimin Zhang, THU-intel at rushes summarization of TRECVID 2008, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.124-128, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463586]
• 34: R. Wright. Personal communication, Technology Manager, Projects, BBC Information & Archives, 2005.
• 35: Koji Yamasaki , Koichi Shinoda , Sadaoki Furui, Automatically estimating number of scenes for rushes summarization, Proceedings of the 2nd ACM TRECVid Video Summarization Workshop, p.129-133, October 31-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1463563.1463587]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Experimentation

Keywords:
benchmarking, evaluation, trecvid, video summarization