Estimation and use of uncertainty in pseudo-relevance feedback

ABSTRACT

Existing pseudo-relevance feedback methods typically perform averaging over the top-retrieved documents, but ignore an important statistical dimension: the risk or variance associated with either the individual document models, or their combination. Treating the baseline feedback method as a black box, and the output feedback model as a random variable, we estimate a posterior distribution for the feed-back model by resampling a given query's top-retrieved documents, using the posterior mean or mode as the enhanced feedback model. We then perform model combination over several enhanced models, each based on a slightly modified query sampled from the original query. We find that resampling documents helps increase individual feedback model precision by removing noise terms, while sampling from the query improves robustness (worst-case performance) by emphasizing terms related to multiple query aspects. The result is a meta-feedback algorithm that is both more robust and more precise than the original strong baseline method.

REFERENCES

• 1: The Lemur toolkit for language modeling and retrieval. http://www.lemurproject.org.
• 2: G. Amati, C. Carpineto, and G. Romano. Query difficulty, robustness, and selective application of query expansion. In Proc. of the 25th European Conf. on Information Retrieval (ECIR 2004), pages 127--137.
• 3: Rie Kubota Ando , Tong Zhang, A high-performance semi-supervised learning method for text chunking, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.1-9, June 25-30, 2005, Ann Arbor, Michigan [doi>10.3115/1219840.1219841]
• 4: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350]
• 5: Claudio Carpineto , Giovanni Romano , Vittorio Giannini, Improving retrieval feedback with multiple term-ranking function combination, ACM Transactions on Information Systems (TOIS), v.20 n.3, p.259-290, July 2002 [doi>10.1145/568727.568728]
• 6: K. Collins-Thompson, P. Ogilvie, and J. Callan. Initial results with structured queries and language models on half a terabyte of text. In Proc. of 2005 Text REtrieval Conference. NIST Special Publication.
• 7: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 8: Warren R. Greiff , William T. Morgan , Jay M. Ponte, The role of variance in term weighting for probabilistic information retrieval, Proceedings of the eleventh international conference on Information and knowledge management, November 04-09, 2002, McLean, Virginia, USA [doi>10.1145/584792.584836]
• 9: T. Kohonen, J. Hynninen, J. Kangas, and J. Laaksonen. SOMPAK: The self-organizing map program package. Technical Report A31, Helsinki University of Technology, 1996. http://www.cis.hut.fi/research/papers/som tr96.ps.Z.
• 10: Victor Lavrenko , W. Bruce Croft , James Allan, A generative theory of relevance, University of Massachusetts Amherst, 2004
• 11: Chi-Hoon Lee , Russ Greiner , Shaojun Wang, Using query-specific variance estimates to combine Bayesian classifiers, Proceedings of the 23rd international conference on Machine learning, p.529-536, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143911]
• 12: Donald Metzler , W. Bruce Croft, Combining the language model and inference network approaches to retrieval, Information Processing and Management: an International Journal, v.40 n.5, p.735-750, September 2004 [doi>10.1016/j.ipm.2004.05.001]
• 13: T. Minka. Estimating a Dirichlet distribution. Technical report, 2000. http://research.microsoft.com/minka/papers/dirichlet.
• 14: J. Ponte. Advances in Information Retrieval, chapter Language models for relevance feedback, pages 73--96. 2000. W. B. Croft, ed.
• 15: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 16: J. Rocchio. The SMART Retrieval System, chapter Relevance Feedback in Information Retrieval, pages 313--323. Prentice-Hall, 1971. G. Salton, ed.
• 17: Tetsuya Sakai , Toshihiko Manabe , Makoto Koyama, Flexible pseudo-relevance feedback via selective sampling, ACM Transactions on Asian Language Information Processing (TALIP), v.4 n.2, p.111-135, June 2005 [doi>10.1145/1105696.1105699]
• 18: Tao Tao , ChengXiang Zhai, Regularized estimation of mixture models for robust pseudo-relevance feedback, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148201]
• 19: Jinxi Xu , W. Bruce Croft, Improving the effectiveness of information retrieval with local context analysis, ACM Transactions on Information Systems (TOIS), v.18 n.1, p.79-112, Jan. 2000 [doi>10.1145/333135.333138]
• 20: Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076121]
• 21: Yun Zhou , W. Bruce Croft, Ranking robustness: a novel framework to predict query performance, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183696]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Algorithms, Experimentation

Keywords:
pseudo-relevance feedback, query expansion