Parsimonious language models for information retrieval

ABSTRACT

We systematically investigate a new approach to estimating the parameters of language models for information retrieval, called parsimonious language models. Parsimonious language models explicitly address the relation between levels of language models that are typically used for smoothing. As such, they need fewer (non-zero) parameters to describe the data. We apply parsimonious models at three stages of the retrieval process: 1) at indexing time; 2) at search time; 3) at feedback time. Experimental results show that we are able to build models that are significantly smaller than standard models, but that still perform at least as well as the standard approaches.

REFERENCES

• 1: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 2: A.P. Dempster, N.M. Laird, and D.B. Rubin. Maximum likelihood from incomplete data via the EM-algorithm plus discussions on the paper. Journal of the Royal Statistical Society, 39(B):1--38, 1977.
• 3: Marcello Federico , Nicola Bertoldi, Statistical cross-language information retrieval using n-best query translations, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564407]
• 4: Djoerd Hiemstra , Franciska de Jong, Disambiguation Strategies for Cross-Language Information Retrieval, Proceedings of the Third European Conference on Research and Advanced Technology for Digital Libraries, p.274-293, September 22-24, 1999
• 5: D. Hiemstra and W. Kraaij. Twenty-One at TREC-7: Ad-hoc and cross-language track. In Proceedings of the seventh Text Retrieval Conference (TREC-7), pages 227--238. NIST Special Publication 500-242, 1998.
• 6: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]
• 7: H. Jin, R. Schwartz, S. Sista, and F. Walls. Topic tracking for radio, TV broadcast and newswire. In Proceedings of the DARPA Broadcast News Workshop, pages 199--204, 1999.
• 8: Wessel Kraaij , Thijs Westerveld , Djoerd Hiemstra, The Importance of Prior Probabilities for Entry Page Search, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564383]
• 9: John Lafferty , Chengxiang Zhai, Document language models, query models, and risk minimization for information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.111-119, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383970]
• 10: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 11: V. Lavrenko and W.B. Croft. Relevance models in information retrieval. In W.B. Croft and J. Lafferty, editors, Language Modeling for Information Retrieval, pages 11--56. Kluwer Academic Publishers, 2003.
• 12: H.P. Luhn. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2:159--165, 1958.
• 13: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 14: David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
• 15: Stefano Mizzaro, Relevance: the whole history, Journal of the American Society for Information Science, v.48 n.9, p.810-832, Sept. 1997 [doi>10.1002/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U]
• 16: K. Ng. A maximum likelihood ratio information retrieval model. In Proceedings of the eighth Text Retrieval Conference (TREC-8). NIST Special Publication 500-246, pages 483--492, 1999.
• 17: J.M. Ponte. Language models for relevance feedback. In W.B. Croft, editor, Advances in information retrieval: recent research from the Center for Intelligent Information Retrieval, pages 73--95. Dordrecht: Kluwer, 2000.
• 18: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 19: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 20: S.E. Robertson and K. Sparck-Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129--146, 1976.
• 21: Ananth Sankar , Venkata Ramana Rao Gadde , Andreas Stolcke , Fuliang Weng, Improved modeling and efficiency for automatic transcription of Broadcast News, Speech Communication, v.37 n.1-2, p.133-158, May 2002 [doi>10.1016/S0167-6393(01)00063-2]
• 22: T. Saracevic. Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26:321--343, 1975.
• 23: Fei Song , W. Bruce Croft, A general language model for information retrieval, Proceedings of the eighth international conference on Information and knowledge management, p.316-321, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320022]
• 24: K. Sparck-Jones, S.E. Robertson, D. Hiemstra, and H. Zaragoza. Language modelling and relevance. In W.B. Croft and J. Lafferty, editors, Language Modeling for Information Retrieval, pages 57--71. Kluwer Academic Publishers, 2003.
• 25: A. Stolcke. Entropy-based pruning of back-off language models. In Proceedings of DARPA Broadcast News Transcription and Understanding Workshop, pages 270--274, 1998.
• 26: Nuno Miguel Borges De Pinho Cruz De Vasconcelos , Andrew B. Lippman, Bayesian models for visual information retrieval, 2000
• 27: E.M. Voorhees. Overview of TREC 2002. In Proceedings of the 11th Text Retrieval Conference (TREC 2002), NIST Special Publication 500-251, pages 1--15, 2002.
• 28: T. Westerveld and A.P. de Vries. Multimedia Retrieval using Multiple Examples. In International Conference on Image and Video Retrieval (CIVR'04), 2004.
• 29: Jinxi Xu , Ralph Weischedel , Chanh Nguyen, Evaluating a probabilistic model for cross-lingual information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.105-110, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383968]
• 30: Chengxiang Zhai , John Lafferty, Model-based feedback in the language modeling approach to information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502654]
• 31: Yi Zhang , Jamie Callan , Thomas Minka, Novelty and redundancy detection in adaptive filtering, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564393]
• 32: Y. Zhang, W. Xu, and J. Callan. Exact maximum likelihood estimation for word mixtures. In Text Learning Workshop in International Conference on Machine Learning (ICML'02), 2002.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation, Theory

Keywords:
information retrieval, language models, parameter estimation