Automatic discovery of query-class-dependent models for multimodal search

ABSTRACT

We develop a framework for the automatic discovery of query classes for query-class-dependent search models in multimodal retrieval. The framework automatically discovers useful query classes by clustering queries in a training set according to the performance of various unimodal search methods, yielding classes of queries which have similar fusion strategies for the combination of unimodal components for multimodal search. We further combine these performance features with the semantic features of the queries during clustering in order to make discovered classes meaningful. The inclusion of the semantic space also makes it possible to choose the correct class for new, unseen queries, which have unknown performance space features. We evaluate the system against the TRECVID 2004 automatic video search task and find that the automatically discovered query classes give an improvement of 18% in MAP over hand-defined query classes used in previous works. We also find that some hand-defined query classes, such as "Named Person" and "Sports" do, indeed, have similarities in search method performance and are useful for query-class-dependent multimodal search, while other hand-defined classes, such as "Named Object" and "General Object" do not have consistent search method performance and should be split apart or replaced with other classes. The proposed framework is general and can be applied to any new domain without expert domain knowledge.

REFERENCES

• 1: LingPipe Named Entity Tagger. Available at: http://www.alias-i.com/lingpipe/. 2004.
• 2: J. Adcock, A. Girgensohn, M. Cooper, T. Liu, L. Wilcox, and E. Rieffel. FXPAL experiments for TRECVID 2004. In TRECVID 2004 Workshop, 2004.
• 3: T.-S. Chua, S.-Y. Neo, K.-Y. Li, G. Wang, R. Shi, M. Zhao, and H. Xu. TRECVID 2004 search and feature extraction task by NUS PRIS. In TRECVID 2004 Workshop, 2004.
• 4: T. G. Dietterich and G. Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263--286, 1995.
• 5: Jean-Luc Gauvain , Lori Lamel , Gilles Adda, The LIMSI Broadcast News transcription system, Speech Communication, v.37 n.1-2, p.89-108, May 2002 [doi>10.1016/S0167-6393(01)00061-9]
• 6: R. Kondor and T. Jebara. A kenel between sets of vectors. In International Conference on Machine Learning, 2003.
• 7: C. Leslie and R. Kuang. Fast kernels for inexact string matching. In Conference on Learning Theory and Kernel Workshop, 2003.
• 8: H. Liu. MontyLingua: An end-to-end natural language processor with common sense. Available at: http://web.media.mit.edu/~hugo/montylingua. 2004.
• 9: G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235--244, 1990.
• 10: M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130--137, July 1980.
• 11: P. Resnik. Using information content to evaluate semantic similarity. In Conference on Artificial Intelligence, 1995.
• 12: S. E. Robertson, S. Walker, M. Hancock-Beaulieu, A. Gull, and M. Lau. Okapi at TREC4. In Text REtrieval Conference, 1992.
• 13: A. Schwaighofer. SVM Toolbox, available at http://www.igi.tugraz.at/aschwaig/software.html. 2002.
• 14: V. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.
• 15: Rong Yan , Jun Yang , Alexander G. Hauptmann, Learning query-class dependent weights in automatic video retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027661]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation, Performance

Keywords:
multimodal fusion, query-class-dependent models, video search