A probabilistic ranking framework using unobservable binary events for video search

ABSTRACT

Recent content-based video retrieval systems combine output of concept detectors (also known as high-level features) with text obtained through automatic speech recognition. This paper concerns the problem of search using the noisy concept detector output only. Unlike term occurrence in text documents, the event of the occurrence of an audiovisual concept is only indirectly observable. We develop a probabilistic ranking framework for unobservable binary events to search in videos, called PR-FUBE. The framework explicitly models the probability of relevance of a video shot through the presence and absence of concepts. From our framework, we derive a ranking formula and show its relationship to previously proposed formulas. We evaluate our framework against two other retrieval approaches using the TRECVID 2005 and 2007 datasets. Especially using large numbers of concepts in retrieval results in good performance. We attribute the observed robustness against the noise introduced by less related concepts to the effective combination of concept presence and absence in our method. The experiments show that an accurate estimate for the probability of occurrence of a particular concept in relevant shots is crucial to obtain effective retrieval results.

REFERENCES

• 1: J. Adcock. Fxpal interacive search experiments for trecvid 2007. In Proceedings of the 7th TRECVID Workshop, Gaithersburg, USA, October 2007.
• 2: S. M. Aji and R. J. McEliece. The generalized distributive law. Information Theory, IEEE Transactions on, 46(2):325--343, 2000.
• 3: R. B. N. Aly, D. Hiemstra, and R. J. F. Ordelman. Building detectors to support searches on combined semantic concepts. In Proceedings of the Multimedia Information Retrieval Workshop, Amsterdam, The Netherlands, pages 40--45, Amsterdam, August 2007. Yahoo! Research.
• 4: A. Budanitsky and G. Hirst. Semantic distance in wordnet: an experimental, application--oriented evaluation of five measures. In In Proceedings of the NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, PA, June. 2001.
• 5: C. Fellbaum. Wordnet: An Electronic Lexical Database. The MIT Press, 1998.
• 6: E. A. Fox and J. A. Shaw. Combination of multiple searches. In TREC, pages 243--252, 1993.
• 7: Norbert Fuhr, Probabilistic models in information retrieval, The Computer Journal, v.35 n.3, p.243-255, June 1992 [doi>10.1093/comjnl/35.3.243]
• 8: J. S. Garofolo, C. Auzanne, and E. M. Voorhees. The trec spoken document retrieval track: A success story. In TREC, 1999.
• 9: C. Hauff, R. B. N. Aly, and D. Hiemstra. The effectiveness of concept based search for video retrieval. In Workshop Information Retrieval (FGIR 2007), Halle, Germany, volume 2007 of LWA 2007 Lernen - Wissen Adaption, pages 205--212, Halle-Wittenberg, 2007. Gesellschaft fuer Informatik.
• 10: D. Hiemstra, H. Rode, R. van Os, and J. Flokstra. Pftijah: text search in an xml database system. In Proceedings of the 2nd International Workshop on Open Source Information Retrieval (OSIR), Seattle, WA, USA, pages 12--17. Ecole Nationale Supérieure des Mines de Saint-Etienne, 2006.
• 11: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]
• 12: Yu-Gang Jiang , Chong-Wah Ngo , Jun Yang, Towards optimal bag-of-features for object categorization and semantic video retrieval, Proceedings of the 6th ACM international conference on Image and video retrieval, p.494-501, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282352]
• 13: L. Kennedy and A. Hauptmann. Lscom lexicon definitions and annotations (version 1.0). Technical report, Columbia University, March 2006.
• 14: T. Mei, X.-S. Hua, W. Lai, L. Yang, Z.-J. Zha, Y. Liu, Z. Gu, G.-J. Qi, M. Wang, J. Tang, X. Yuan, Z. Lu, and J. Liu. Msra-ustc-sjtu at trecvid 2007: High-level feature extraction and search. In Proceedings of the 7th TRECVID Workshop, Gaithersburg, USA, October 2007. To be published.
• 15: L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford Digital Library Technologies Project, 1998.
• 16: S. Robertson. The probability ranking principle in ir. J. Documentation, 33:294--304, 1977.
• 17: Ian Ruthven , Mounia Lalmas, A survey on the use of relevance feedback for information access systems, The Knowledge Engineering Review, v.18 n.2, p.95-145, June 2003 [doi>10.1017/S0269888903000638]
• 18: G. Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1971
• 19: N. Sebe. The state of the art in image and video retrieval. In Image and Video Retrieval, volume Volume 2728/2003, pages 1--8. Springer Berlin / Heidelberg, 2003.
• 20: C. G. M. Snoek, J. C. van Gemert, T. Gevers, B. Huurnink, D. C. Koelma, M. van Liempt, O. de Rooij, K. E. A. van de Sande, F. J. Seinstra, A. W. M. Smeulders, A. H. C. Thean, C. J. Veenman, and M. Worring. The mediamill trecvid 2007 semantic video search engine. In Proceedings of the 7th TRECVID Workshop, Gaithersburg, USA, October 2007. To be published.
• 21: Cees G. M. Snoek , Marcel Worring , Jan C. van Gemert , Jan-Mark Geusebroek , Arnold W. M. Smeulders, The challenge problem for automated detection of 101 semantic concepts in multimedia, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180727]
• 22: K. Sparck Jones , S. Walker , S. E. Robertson, A probabilistic model of information retrieval: development and comparative experiments Part 2, Information Processing and Management: an International Journal, v.36 n.6, p.809-840, Nov.06.2000 [doi>10.1016/S0306-4573(00)00016-9]
• 23: L. van der Werff and W. Heeren. Evaluating asr output for information retrieval. In F. de Jong, D. Oard, R. Ordelman, and S. Raaijmakers, editors, Proceedings of the ACM SIGIR Workshop 'Searching Spontaneous Conversational Speech, Enschede, 2007. CTIT.
• 24: Rong Yan , Alexander G. Hauptmann, Probabilistic models for combining diverse knowledge sources in multimedia retrieval, Carnegie Mellon University, Pittsburgh, PA, 2006
• 25: W. Zheng, J. Li, Z. Si, F. Lin, and B. Zhang. Using high-level semantic features in video retrieval. In Image and Video Retrieval, volume Volume 4071/2006, pages 370--379. Springer Berlin / Heidelberg, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Algorithms, Experimentation, Performance, Theory

Keywords:
concept based search, multimedia information retrieval, probabilistic information retrieval