Analysing the performance of visual, concept and text features in content-based video retrieval

ABSTRACT

This paper describes revised content-based search experiments in the context of TRECVID 2003 benchmark. Experiments focus on measuring content-based video retrieval performance with following search cues: visual features, semantic concepts and text. The fusion of features uses weights and similarity ranks. Visual similarity is computed using Temporal Gradient Correlogram and Temporal Color Correlogram features that are extracted from the dynamic content of a video shot. Automatic speech recognition transcripts and concept detectors enable higher-level semantic searching. 60 hours of news videos from TRECVID 2003 search task were used in the experiments. System performance was evaluated with 25 pre-defined search topics using average precision. In visual search, multiple examples improved the results over single example search. Weighted fusion of text, concept and visual features improved the performance over text search baseline. Expanded query term list of text queries gave also notable increase in performance over the baseline text search

REFERENCES

• 1: IBM CueVideo Toolkit. http://www.almaden.ibm.com/projects/cuevideo.shtml (2.2.2004)
• 2: Howard D. Wactlar , Michael G. Christel , Yihong Gong , Alexander G. Hauptmann, Lessons Learned from Building a Terabyte Digital Video Library, Computer, v.32 n.2, p.66-73, February 1999 [doi>10.1109/2.745722]
• 3: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146]
• 4: Alan F. Smeaton , Hyowon Lee , Kieran McDonald, Experiences of creating four video library collections with the Físchlár system, International Journal on Digital Libraries, v.4 n.1, p.42-44, August 2004 [doi>10.1007/s00799-003-0065-5]
• 5: Hampapur, A., Gupta, A., Horowitz, B., Shu, C.-F., Fuller, C., Bach, J., Gorkani, M. and Jain, R. Virage Video Engine. In Proceedings of SPIE vol. 3022, Storage and Retrieval for Image and Video Databases, 1997, 188--198.
• 6: Naphade, M.R., Kristjansson, T., Frey, B., and Huang, T.S. Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems. In Proceedings of International Conference on Image Processing, vol. 3. 1998, 536--540.
• 7: Semantic Video Indexing Using a Probabilistic Framework, Proceedings of the International Conference on Pattern Recognition, p.3083, September 03-08, 2000
• 8: Chang, S.F., Chen, W., and Sundaram, H. Semantic visual templates - linking features to semantics. In Proceedings of IEEE International Conference on Image Processing, vol. 3., 1998, 531--535.
• 9: Del Bimbo, A. Expressive semantics for automatic annotation and retrieval of video streams. In Proceedings of IEEE International Conference on Multimedia and Expo, Vol.2., 2000, 671--674.
• 10: Adams, W., Iyengar, G., Lin, C., Naphade, M., Neti, C., Nock, H., and Smith, J. Semantic Indexing of Multimedia Content using Visual, Audio and Text Cues. EURASIP Journal on Applied Signal Processing, vol. 2, 2003, 1--16.
• 11: TREC Video Retrieval Evaluation. http://www-nlpir.nist.gov/projects/trecvid/ (4.6.2004)
• 12: Mika Rautiainen , Davis Doermann, Temporal Color Correlograms for Video Retrieval, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 1, p.10267, August 11-15, 2002
• 13: Prewitt, J.M.S. Object enhancement and extraction. In B.S.Lipkin and A. Rosenfeld, (eds) Picture Processing and Psychopictorics, Academic Press, New York, 1970.
• 14: Jing Huang , S. Ravi Kumar , Mandar Mitra , Wei-Jing Zhu , Ramin Zabih, Image Indexing Using Color Correlograms, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.762, June 17-19, 1997
• 15: B.S. Manjunath , Phillipe Salembier , Thomas Sikora, Introduction to MPEG-7: Multimedia Content Description Interface, John Wiley & Sons, Inc., New York, NY, 2002
• 16: Tin Kam Ho , Jonathan J. Hull , Sargur N. Srihari, Decision Combination in Multiple Classifier Systems, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.16 n.1, p.66-75, January 1994 [doi>10.1109/34.273716]
• 17: Rautiainen, M., Seppänen, T., Penttilä, J., and Peltola, J. Detecting semantic concepts from video using temporal gradients and audio classification. In Proceedings of International Conference on Image and Video Retrieval, Urbana, IL, 2003, 260--270.
• 18: Salton, G. and Yang, C. On the specification of term values in automatic indexing. Journal of Documentation, Vol. 29, 1973, 351--372.
• 19: Porter, M. An Algorithm for Suffix Stripping Program. Program, 14(3), 1980, 130--137.
• 20: IBM VideoAnnEx Server Page MPEG-7 (4.2.2004) http://mp7.watson.ibm.com/VideoAnnEx/
• 21: Rautiainen, M., Penttilä, J., Pietarila, P., Noponen, K., Hosio, M., Koskela, T., Mäkelä, S.M., Peltola, J., Liu, J., Ojala, T., and Seppänen, T. TRECVID 2003 experiments at MediaTeam Oulu and VTT. TRECVID Workshop at Text Retrieval Conference TREC-2003, Gaithersburg, MD, 2003.
• 22: Jean-Luc Gauvain , Lori Lamel , Gilles Adda, The LIMSI Broadcast News transcription system, Speech Communication, v.37 n.1-2, p.89-108, May 2002 [doi>10.1016/S0167-6393(01)00061-9]
• 23: Rautiainen, M., Ojala, T. and Seppänen T. Cluster-temporal video browsing with semantic filtering. In Proceedings of Advanced Concepts for Intelligent Vision Systems, Ghent, Belgium, 2003, 116--123.
• 24: NIST: Common Evaluation Measures. appendix in Special Publication 500-250 (TREC 2001), NIST, Gaithersburg, MD, http://trec.nist.gov/pubs/trec10/appendices/measures.pdf.
• 25: Text REtrieval Conference TREC. http://trec.nist.gov/ (26.3.2004)
• 26: Rautiainen, M., Ojala, T., and Seppänen, T. Cluster-temporal browsing of large news video databases. in Proceedings of 2004 IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Retrieval models

General Terms:
Algorithms, Experimentation, Performance

Keywords:
Borda count, content-based video retrieval, feature fusion, semantic concepts