Statistical cross-language information retrieval using n-best query translations

ABSTRACT

This paper presents a novel statistical model for cross-language information retrieval. Given a written query in the source language, documents in the target language are ranked by integrating probabilities computed by two statistical models: a query-translation model, which generates most probable term-by-term translations of the query, and a query-document model, which evaluates the likelihood of each document and translation. Integration of the two scores is performed over the set of N most probable translations of the query. Experimental results with values N=1, 5, 10 are presented on the Italian-English bilingual track data used in the CLEF 2000 and 2001 evaluation campaigns.

REFERENCES

• 1: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 2: Nicola Bertoldi , Marcello Federico, ITC-irst at CLEF 2000: Italian Monolingual Track, Revised Papers from the Workshop of Cross-Language Evaluation Forum on Cross-Language Information Retrieval and Evaluation, p.261-272, September 21-22, 2000
• 3: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 4: William B. Frakes , Ricardo Baeza-Yates, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 5: Djoerd Hiemstra , Franciska de Jong, Disambiguation Strategies for Cross-Language Information Retrieval, Proceedings of the Third European Conference on Research and Advanced Technology for Digital Libraries, p.274-293, September 22-24, 1999
• 6: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 7: S. Johnson, P. Jourlin, K. S. Jones, and P. Woodland. Spoken document retrieval for TREC-8 at Cambridge University. In Proceedings of the 8th Text REtrieval Conference, Gaithersburg, MD, 1999.
• 8: W. Kraaij, R. Pohlmann, and D. Hiemstra. Twenty-One at TREC-8: using Language Technology for Information Retrieval. In Proceedings of the 8th Text Retrieval Conference TREC-8, pages 285--300, 2000.
• 9: D. R. H. Miller, T. Leek, and R. M. Schwartz. BBN at TREC-7: Using hidden Markov models for information retrieval. In Proceedings of the 7th Text REtrieval Conference, pages 133--142, Gaithersburg, MD, 1998.
• 10: H. Ney, U. Essen, and R. Kneser. On structuring probabilistic dependences in stochastic language modelling. Computer Speech and Language, 8:1--38, 1994.
• 11: K. Ng. A maximum likelihood ratio information retrieval model. In Proceedings of the 8th Text REtrieval Conference, Gaithersburg, MD, 1999.
• 12: N. J. Nilsson, Principles of artificial intelligence, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1980
• 13: Revised Papers from the Workshop of Cross-Language Evaluation Forum on Cross-Language Information Retrieval and Evaluation, September 2000
• 14: C. Peters, editor. Working notes for the CLEF 2001 Workshop. Darmstatd, Germany, 2001.
• 15: Lawrence R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, Readings in speech recognition, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1990
• 16: F. K. Soong and E. F. Huang. A tree-trellis based fast search for finding the n-best sentence hypotheses in continuous speech recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 705--708, Toronto, Canada, 1991.
• 17: I. H. Witten and T. C. Bell. The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression. IEEE Trans. Inform. Theory, IT-37(4):1085--1094, 1991.
• 18: Jinxi Xu , Ralph Weischedel , Chanh Nguyen, Evaluating a probabilistic model for cross-lingual information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.105-110, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383968]
• 19: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Theory

Keywords:
CLIR, query translation, statistical model