Evaluation campaigns and TRECVid

ABSTRACT

The TREC Video Retrieval Evaluation (TRECVid)is an international benchmarking activity to encourage research in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations 1 interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video corpus,automatic detection of a variety of semantic and low-level video features, shot boundary detection and the detection of story boundaries in broadcast TV news. This paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, high-lighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation bench-marking campaign and this allows us to discuss whether such campaigns are a good thing or a bad thing. There are arguments for and against these campaigns and we present some of them in the paper concluding that on balance they have had a very positive impact on research progress.

REFERENCES

• 1: AMI: Augmented Multi-Person Interaction. URL:www.amiproject.org/,Last checked 21 June 2006.
• 2: ARGOS: Evaluation Campaign for Surveillance Tools of Video Content. URL: www.irit.fr/recherches/SAMOVA/MEMBERS/-JOLY/argos/, Last checked 21 June 2006.
• 3: CLEAR '06 Evaluation Campaign and Workshop - Classification of Events, Activities and Relationships. URL:www.clear-evaluation.org/,Last checked 21 June 2006.
• 4: ETISEO: Video Understanding Evaluation. URL:www.silogic.fr/etiseo/, June 2006.
• 5: Face Recognition Grand Challenge. URL:www.frvt.org/FRGC,2006.
• 6: INEX:INitiative for the Evaluation of XML Retrieval. URL:inex.is.informatik.uni-duisburg.de/,Last checked 21 June 2006.
• 7: The Internet Archive Movie Archive home page. URL:www.archive.org/movies, 2006.
• 8: Lscom lexicon definitions and annotations. URL:www.ee.columbia.edu/dvmm/lscom, 2006.
• 9: NTCIR:NII Test Collection for IR Systems Project. URL:research.nii.ac.jp/ntcir/, Last checked June 2006.
• 10: PETS 2006: Ninth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance. URL:www.pets2006.net/,Last checked 21 June 2006.
• 11: The Benchathlon Network: Home of CBIR Benchmarking. URL:www.benchathlon.net/, Last checked 21 June 2006.
• 12: The Cross-Language Evaluation Forum (CLEF). URL:clef.isti.cnr.it/, Last checked 21 June 2006.
• 13: The IMAG-EVAL Evaluation Campaign. URL:www.imageval.org/, Last checked 21 June 2006.
• 14: M. G. Christel and A. G. Hauptmann. The Use and Utility of High-Level Semantic Features in Video Retrieval. In Proceedings of the International Conference on Video Retrieval pages 134--144, Singapore, 20-22 July 2005.
• 15: Norbert Fuhr , Mounia Lalmas, Introduction to the Special Issue on INEX, Information Retrieval, v.8 n.4, p.515-519, December 2005 [doi>10.1007/s10791-005-0745-4]
• 16: Alexander G. Hauptmann , Michael G. Christel, Successful approaches in the TREC video retrieval evaluations, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027681]
• 17: Peter Ingwersen , Kalervo Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 18: B. F. J. Manly. Randomization, Bootstrap, and Monte Carlo Methods in Biology Chapman & Hall, London, UK, 2nd edition, 1997.
• 19: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 20: NIST. TREC Video Retrieval Evaluation Publications. URL: www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html,2006.
• 21: R. Ruiloba, P. Joly, S. Marchand-Maillet, and G. Quénot. Towards a Standard Protocol for the Evaluation of Video-to-Shots Segmentation Algorithms. In European Workshop on Content Based Multimedia Indexing Toulouse, France, October 1999.
• 22: Cees G. M. Snoek , Marcel Worring , Arnold W. M. Smeulders, Early versus late fusion in semantic video analysis, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101236]
• 23: Cees G. M. Snoek , Marcel Worring , Jan-Mark Geusebroek , Dennis C. Koelma , Frank J. Seinstra , Arnold W. M. Smeulders, The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.10, p.1678-1689, October 2006 [doi>10.1109/TPAMI.2006.212]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Evaluation/methodology

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
benchmarking, evaluation, video retrieval