A general language model for information retrieval

ABSTRACT

Statistical language modeling has been successfully used for speech recognition, part-of-speech tagging, and syntactic parsing. Recently, it has also been applied to information retrieval. According to this new paradigm, each document is viewed as a language sample, and a query as a generation process. The retrieved documents are ranked based on the probabilities of producing a query from the corresponding language models of these documents. In this paper, we will present a new language model for information retrieval, which is based on a range of data smoothing techniques, including the Good-Turning estimate, curve-fitting functions, and model combinations. Our model is conceptually simple and intuitive, and can be easily extended to incorporate probabilities of phrases such as word pairs and word triples. The experiments with the Wall Street Journal and TREC4 data sets showed that the performance of our model is comparable to that of INQUERY and better than that of another language model for information retrieval. In particular, word pairs are shown to be useful in improving the retrieval performance.

REFERENCES

• 1: James P. Callan , W. Bruce Croft , John Broglio, TREC and TIPSTER experiments with INQUERY, Information Processing and Management: an International Journal, v.31 n.3, p.327-343, May-June 1995
• 2: Chamiak, E. Statistical Language Learning. The M1T Press, Cambridge MA, 1993.
• 3: W. Bruce Croft , Howard R. Turtle, Text retrieval and inference, Text-based intelligent systems: current research and practice in information extraction and retrieval, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1992
• 4: Fralces, W.B., and Baeza-Yates, R. (editors). Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, New Jersey: Prentice Hall, 1992.
• 5: Djoerd Hiemstra, A Linguistically Motivated Probabilistic Model of Information Retrieval, Proceedings of the Second European Conference on Research and Advanced Technology for Digital Libraries, p.569-584, September 21-23, 1998
• 6: Leek, T., Miller, D.R.H., and Schwartz, R.M. A Hidden Markov Model Information Retrieval System. TREC-7 PrOngs, 1998.
• 7: Christopher D. Manning , Hinrich Schütze, Foundations of statistical natural language processing, MIT Press, Cambridge, MA, 1999
• 8: David R. H. Miller , Tim Leek , Richard M. Schwartz, A hidden Markov model information retrieval system, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.214-221, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312680]
• 9: Jay Michael Ponte, A language modeling approach to information retrieval, University of Massachusetts, Amherst, MA, 1998
• 10: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 11: Robcrtson, S.E. The probability ranking principle in IR. journal of Documentation, 33(4): 294-304, Decem~r 1977.
• 12: Gerard. Salton, Automatic Information Organization and Retrieval., McGraw Hill Text, 1968

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT

General Terms:
Design, Documentation, Experimentation, Languages, Management, Measurement, Performance, Theory

Keywords:
curve-fitting functions, good-turing estimate, model combinations, statistical language modeling