Successful approaches in the TREC video retrieval evaluations

ABSTRACT

This paper reviews successful approaches in evaluations of video retrieval over the last three years. The task involves the search and retrieval of shots from MPEG digitized video recordings using a combination of automatic speech, image and video analysis and information retrieval technologies. The search evaluations are grouped into interactive (with a human in the loop) and non-interactive (where the human merely enters the query into the system) submissions. Most non-interactive search approaches have relied extensively on text retrieval, and only recently have image-based features contributed reliably to improved search performance. Interactive approaches have substantially outperformed all non-interactive approaches, with most systems relying heavily on the user's ability to refine queries and reject spurious answers. We will examine both the successful automatic search approaches and the user interface techniques that have enabled high performance video retrieval.

REFERENCES

• 1: Boldareva, L., de Vries, A., and Hiemstra, D. Monitoring User-System Performance in Interactive Retrieval Tasks. Proc. RIAO 2004 (Avignon, France, April 2004), pp. 474--483.
• 2: Boldareva, L., and Hiemstra, D. Interactive Content-Based Retrieval Using Pre-computed Object-Object Similarities. In Proc. CIVR 2004 (Dublin, Ireland, July 2004).
• 3: Christel, M., Huang, C., Moraveji, N., and Papernick, N. A Comparative Study of Evidence Combination Strategies. In Proc. ICASSP (Montreal, May 2004), pp. 1032--1035.
• 4: Michael G. Christel , Neema Moraveji , Chang Huang, Evaluating content-based filters for image and video retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009135]
• 5: Christel, M. submitted to ACM Multimedia 2004.
• 6: Peter Enser , Christine Sandom, Retrieval of Archival Moving Imagery - CBIR Outside the Frame?, Proceedings of the International Conference on Image and Video Retrieval, p.206-214, July 18-19, 2002
• 7: Jean-Luc Gauvain , Lori Lamel , Gilles Adda, The LIMSI Broadcast News transcription system, Speech Communication, v.37 n.1-2, p.89-108, May 2002 [doi>10.1016/S0167-6393(01)00061-9]
• 8: Lee, H. and Smeaton, A.F. Designing the User Interface for the Fischlar Digital Video Library. J. Digital Info 2(4), May 2002.
• 9: NIST, Digital Video Retrieval at NIST: TREC Video Retrieval Evaluation, 2001-2004, http://www-nlpir.nist.gov/projects/trecvid/.
• 10: Rautiainen, M., Ojala, T., and Seppanen, T. Cluster-temporal browsing of large news video databases. In Proc. IEEE ICME (Taipei, Taiwan, June 2004).
• 11: Rowe, L.A. and Jain, R. ACM SIGMM Retreat Report on Future Directions in Multimedia Research, March 2004, www.acm.org/sigmm/main/events/sigmm_retreat/sigmm-retreat03-final.htm.
• 12: Thijs Westerveld , Arjen P. de Vries, Experimental result analysis for a generative probabilistic image retrieval model, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860461]
• 13: Wildemuth, B., Yang, M., Hughes, A., Gruss, R., Geisler, G., and Marchionini, G. Access via Features vs. Access via Transcripts: User Performance and Satisfaction. In Proc. TRECVID (Gaithersburg, MD, Nov. 2003),
• 14: Yavlinsky, A., Pickering, M., Heesch, D., Ruger, S. A Comparative Study of Evidence Combination Strategies. In Proc. ICASSP (Montreal, May 2004), pp. 1040--1043.
• 15: Alan F. Smeaton , Paul Over , Cash Costello , Arjen P. de Vries , David S. Doermann , Alexander G. Hauptmann , Mark E. Rorvig , John R. Smith , Lide Wu, The TREC2001 Video Track: Information Retrieval on Digital Video Information, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.266-275, September 16-18, 2002
• 16: R. Ruiloba, P. Joly, S. Marchand-Maillet, G. Quénot : "Towards a Standard Protocol for the Evaluation of Video-to-Shots Segmentation Algorithms", International Workshop in Content-Based Multimedia Indexing (CBMI), Toulouse France.
• 17: NIST TREC 2002, Results of the Video Track, http://trec.nist.gov/pubs/trec10/appendices/video_results.html
• 18: J. Smith, S. Srinivasan, A. Amir, S. Basu, G. Iyengar, C. Lin, Milind Naade, D. Ponceleon, and B Tseng, "Integrating Features, Models, and Semantics for TREC Video Retrieval," NIST TREC-10 Text Retrieval Conference, Gaithersburg, Maryland, November 2001.
• 19: J. Baan, A. van Ballegooij, J-M. Geusebroek, D. Hiemstra, J. den Hartog, J. List, C. Snoek, I. Patras, S. Raaijmakers, L. Todoran, J. Vendrig, A. de Vries, T. Westerveld and M. Worring, Lazy Users and Automatic Video Retrieval Tools in (the) Lowlands. In Proceedings of the 10th Text Retrieval Conference (TREC), November 2001.
• 20: Hauptmann, A., Jin, R. , N. Papernick, D. Ng, Y. Qi, Houghton, R Thornton, S. Video Retrieval with the Informedia Digital Video Library System , The Tenth Text Retrieval Conference (TREC-2001) Gaithersburg, Maryland, November 13-16, 2001
• 21: J. R. Smith, W.H. Adams, A. Amir, C. Dorai, S. Ghosal, G. Iyengar, A. Jaimes, C. Lang, C.-Y.Lin, A. Natsev, C. Neti, H. J. Nock, H. Permuter, R. Singh, S. Srinivasan, B. L. Tseng, AT Varadaraju, D. Zhang,"IBM Research TREC-2002 Video Retrieval System," NIST Text Retrieval Conference (TREC-2002), Nov., 2002.
• 22: J. Vendrig, J. den Hartog, D. van Leeuwen, I. Patras, S. Raaijmakers, C. Snoek, J. van Rest and M. Worring, TREC Feature Extraction by Active Learning. In Proceedings of the 11th Text Retrieval Conference (TREC), Nov. 2002.
• 23: X.-S. Hua, P. Yin, H. Wang, J. Chen, L. Lu, M. Li, H.-J. Zhang, "MSR-Asia at TREC-11 Video Track," TRECVID 2002.
• 24: Rautiainen M, Penttilä J, Vorobiev D, Noponen K, Väyrynen P, Hosio M, Matinmikko E, Mäkelä SM, Peltola J, Ojala T & Seppäänen T (2002) TREC 2002 Video Track experiments at MediaTeam Oulu and VTT. Proc. Text Retrieval Conference TREC 2002 Video Track, Gaithersburg, MD, Nov. 2002.
• 25: Wolf, C., Doermann, D., and Rautiainen, M. Video Indexing and Retrieval at UMD. Proceedings of the Text Retrieval Conference (TREC) 2002. November 19th-22th, 2002, Gaithersburg, MD, USA
• 26: A. Hauptmann, R.V. Baron, M.-Y. Chen, M. Christel, P. Duygulu, C. Huang, R. Jin, W.-H. Lin, T. Ng, N. Moraveji, N. Papernick, C.G.M. Snoek, G. Tzanetakis, J. Yang, R. Yan, and H.D. Wactlar, Informedia at TRECVID 2003: Analyzing and Searching Broadcast News Video, Proceedings of (VIDEO) TREC 2003, November 2003.
• 27: Chen, M-Y., and Hauptmann, A., Searching for a Specific Person in Broadcast News Video, International Conference on Acoustics, Speech, and Signal Processing (ICASSP'04), Montreal, Canada, May 17-21, 2004
• 28: Westerveld, T,. Ianeva, T., Boldareva, L., de Vries, A.P. and Hiemstra, D. Combining Information Sources for Video Retrieval In: TRECVID 2003 Workshop, Nov., 2003
• 29: A. Amir, W, Hsu, G. Iyengar, C.-Y.Lin, M. Naade, A. Natsev, C. Neti, H. J. Nock, J. R. Smith, B. L. Tseng, Y. Wu, D. Zhang, "IBM Research TRECVID-2003 System," Proc. NIST Text Retrieval Conf. (TREC), Gaithersburg, MD, Nov., 2003.
• 30: The Internet Archive Movie Archive Home Page. (2002) URL: www.archive.org/movies
• 31: Open Video Digital Library, http://www.open-video.org/
• 32: Yan, R., Hauptmann, A.G. and Jin, R., Pseudo-Relevance Feedback for Multimedia Retrieval, in Video Mining, Rosenfeld, A., Doermann, D., and DeMenthon, D. (eds), Kluwer, Boston, pp. 309--338, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Experimentation

Keywords:
TRECVID, information retrieval benchmarks, search, video retrieval