Empirical development of an exponential probabilistic model for text retrieval: using textual analysis to build a better model

ABSTRACT

Much work in information retrieval focuses on using a model of documents and queries to derive retrieval algorithms. Model based development is a useful alternative to heuristic development because in a model the assumptions are explicit and can be examined and refined independent of the particular retrieval algorithm. We explore the explicit assumptions underlying the naïve framework by performing computational analysis of actual corpora and queries to devise a generative document model that closely matches text. Our thesis is that a model so developed will be more accurate than existing models, and thus more useful in retrieval, as well as other applications. We test this by learning from a corpus the best document model. We find the learned model better predicts the existence of text data and has improved performance on certain IR tasks.

REFERENCES

• 1: J. M. Bernardo and A. F. M. Smith. Bayesian Theory. John Wiley & Sons, 1994.
• 2: K. Church and W. Gale. Poisson mixtures. Natural Language Engineering, 1(2):163--190, 1995.
• 3: S. Eyheramendy, D. D. Lewis, and D. Madigan. On the naive Bayes model for text classification. In Artificial Intelligence & Statistics, 2003.
• 4: Larry Fitzpatrick , Mei Dent, Automatic feedback using past queries: social searching?, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.306-313, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 5: Norbert Fuhr, Probabilistic models in information retrieval, The Computer Journal, v.35 n.3, p.243-255, June 1992 [doi>10.1093/comjnl/35.3.243]
• 6: A. T. Gous. Adaptive estimation of distributions using exponential sub-families. Journal of Computational and Graphical Statistics, 7(3):388--396, 1998.
• 7: Warren R. Greiff, A theory of term weighting based on exploratory data analysis, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.11-19, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290948]
• 8: A. Griffith, H. C. Luckhurst, and P. Willett. Using interdocument similarity information in document retrieval systems. JASIS, 37:3--11, 1986.
• 9: D. Heckerman. A tutorial on learning with Bayesian networks. Technical Report MSR_TR-95-06, Microsoft Research, 1995. Revised 1996.
• 10: Rong Jin , Alex G. Hauptmann , Cheng Xiang Zhai, Title language model for information retrieval, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564386]
• 11: K. S. Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11--21, 1972.
• 12: K. S. Jones, S. Walker, and S. Robertson. A probabilistic model of information retrieval: development and status. Technical Report TR-446, Cambridge University Computer Laboratory, 1998.
• 13: T. Kalt, A New Probabilistic Model of Text Classification and Retrieval TITLE2:, University of Massachusetts, Amherst, MA, 1998
• 14: Slava M. Katz, Distribution of content words and phrases in text and language modelling, Natural Language Engineering, v.2 n.1, p.15-59, March 1996 [doi>10.1017/S1351324996001246]
• 15: K. L. Kwok , M. Chan, Improving two-stage ad-hoc retrieval for short queries, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.250-256, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291003]
• 16: David D. Lewis, Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval, Proceedings of the 10th European Conference on Machine Learning, p.4-15, April 21-23, 1998
• 17: H. P. Luhn. A statistical approach to mechanized encoding and searching of literary information. IBM Journal of Research and Developement, 1(4):309--317, 1957.
• 18: Kathleen R. McKeown , Judith L. Klavans , Vasileios Hatzivassiloglou , Regina Barzilay , Eleazar Eskin, Towards multidocument summarization by reformulation: progress and prospects, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.453-460, July 18-22, 1999, Orlando, Florida, United States
• 19: K. Ng. A maximum likelihood ratio information retrieval model. In TREC-8, 1999.
• 20: Kamal Nigam , Andrew McCallum , Sebastian Thrun , Tom Mitchell, Learning to classify text from labeled and unlabeled documents, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.792-799, July 1998, Madison, Wisconsin, United States
• 21: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 22: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 23: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 24: ChengXiang Zhai , John Lafferty, Two-stage language models for information retrieval, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564387]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Experimentation

Keywords:
formal models, information retrieval, machine learning