Relevance models to help estimate document and query parameters

ABSTRACT

A central idea of Language Models is that documents (and perhaps queries) are random variables, generated by data-generating functions that are characterized by document (query) parameters. The key new idea of this paper is to model that a relevance judgment is also generated stochastically, and that its data generating function is also governed by those same document and query parameters. The result of this addition is that any available relevance judgments are easily incorporated as additional evidence about the true document and query model parameters. An additional aspect of this approach is that it also resolves the long-standing problem of document-oriented versus query-oriented probabilities. The general approach can be used with a wide variety of hypothesized distributions for documents, queries, and relevance. We test the approach on Reuters Corpus Volume 1, using one set of possible distributions. Experimental results show that the approach does succeed in incorporating relevance data to improve estimates of both document and query parameters, but on this data and for the specific distributions we hypothesized, performance was no better than two separate one-sided models. We conclude that the model's theoretical contribution is its integration of relevance models, document models, and query models, and that the potential for additional performance improvement over one-sided methods requires refinements.

REFERENCES

• 1: A unified maximum likelihood approach to document retrieval, Journal of the American Society for Information Science and Technology, v.52 n.10, p.785-796, August 2001 [doi>10.1002/asi.1137.abs]
• 2: Bodoff, D., Wu, B., and Wong, Ky. 2003. Relevance data for language models using maximum likelihood. J. Amer. Soc. Info. Sci. Tech. 54, 11, 1050--1061.
• 3: Brauen, T. 1971. Document Vector Modification. In The SMART Retrieval System: Experiments in Automatic Document Processing, G. Salton, Ed. Prentice Hall, Engelwood Cliffs, NJ, 456--484.
• 4: Cooper, W. S., Chen, A., and Gey, F. C. 1995. Experiments in the probabilistic retrieval of full text documents. In Proceedings of TREC3, Donna K. Harman, Ed. National Institute of Standards and Technology, Gaithersburg, MD.
• 5: Norbert Fuhr , Chris Buckley, A probabilistic learning approach for document indexing, ACM Transactions on Information Systems (TOIS), v.9 n.3, p.223-248, July 1991 [doi>10.1145/125187.125189]
• 6: Fuhr, N., Pfeifer, U., Bremkamp, C., and Pollmann, M. 1994. Probabilistic learning approaches for indexing and retrieval with the TREC-2 collection. In Proceedings of TREC3, Donna K. Harman, Ed. National Institute of Standards and Technology, Gaithersburg, MD.
• 7: Hansen, L. K. and Larsen, J. 1996. Linear unlearning for cross-validation. Advances in Computational Mathematics 5, 269--280.
• 8: John Lafferty , Chengxiang Zhai, Document language models, query models, and risk minimization for information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.111-119, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383970]
• 9: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 10: Nie, J. Y. 1989. An Outline of a general model for information retrieval. Information Processing and Management 25, 5, 477--491.
• 11: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 12: S. E. Robertson, On term selection for query expansion, Journal of Documentation, v.46 n.4, p.359-364, Dec. 1990 [doi>10.1108/eb026866]
• 13: Robertson, S. E. 1994. Query-document symmetry and dual models. J. Documentation 50, 3, 233--238.
• 14: Robertson, S. E. and Sparck Jones, K. 1976. Relevance weighting of search terms. J. Amer. Soc. Info. Sci. 27, 3, 129--146.
• 15: Robertson, S. E. Maron, M. E., and Cooper, W. S. 1982. Probability of relevance: a unification of two competing models for document retrieval. Information Technology---Research and Development 1, 1--21.
• 16: Hinrich Schütze , David A. Hull , Jan O. Pedersen, A comparison of classifiers and document representations for the routing problem, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.229-237, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215365]
• 17: Jaime Teevan , David R. Karger, Empirical development of an exponential probabilistic model for text retrieval: using textual analysis to build a better model, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860441]
• 18: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997
• 19: Chengxiang Zhai , John Lafferty, Model-based feedback in the language modeling approach to information retrieval, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502654]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Theory

Keywords:
Probabilistic models, language models