Ranking robustness: a novel framework to predict query performance

ABSTRACT

In this paper, we introduce the notion of ranking robustness, which refers to a property of a ranked list of documents that indicates how stable the ranking is in the presence of uncertainty in the ranked documents. We propose a statistical measure called the robustness score to quantify this notion. We demonstrate that the robustness score significantly and consistently correlates with query performance in a variety of TREC test collections including the GOV2 collection. We compare the robustness score with the clarity score method which is the state-of-the-art technique for query performance prediction. Our experimental results show that the robustness score performs better than or at least as good as the clarity score. We find that the clarity score is barely correlated with query performance on the GOV2 collection while the correlation between the robustness score and query performance remains significant. We also notice that a combination of the two usually results in more prediction power.

REFERENCES

• 1: E.M. Voorhees. Overview of the TREC 2004 Robust Track. In the Online Proceedings of 2004 Text REtrieval Conference (TREC 2004).
• 2: Robust Track http://trec.nist.gov/tracks.html.
• 3: Predicting Query Difficulty. SIGIR workshop 2005 http://www.haifa.ibm.com/sigir05-qp/index.html
• 4: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 5: S. Tomlinson. Robust, Web and Terabyte Retrieval with Hummingbird SearchServer at TREC 2004. In the Online Proceedings of 2004 Text REtrieval Conference (TREC 2004).
• 6: B. He and I. Ounis. Inferring query performance using pre-retrieval predictors. In proceedings of the SPIRE 2004. pp 43--54, 2004.
• 7: V. Plachouras, B. He, I. Ounis. University of Glasgow at TREC2004: Experiments in Web, Robust, and Terabyte Tracks with Terrier. In the Online Proceedings of 2004 Text REtrieval Conference (TREC 2004).
• 8: Giambattista Amati, Claudio Carpineto, and Giovanni Romano. Query difficulty, robustness and selective application of query expansion In ECIR 2004, pp 127--137.
• 9: Fernando Diaz , Rosie Jones, Using temporal profiles of queries for precision prediction, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1008998]
• 10: K. L. Kwok, L. Grunfeld, N. Dinstl, P. Deng. TREC 2005 Robust Track Experiments Using PIRCS. In the Online Proceedings of 2005 Text REtrieval Conference (TREC 2005).
• 11: Y. Bernstein, B. Billerbeck, S. Garcia, N. Lester, F. Scholer, J. Zobe. RMIT University at TREC 2005: Terabyte and Robust Track. In the Online Proceedings of 2005 Text REtrieval Conference (TREC 2005).
• 12: Elad Yom-Tov , Shai Fine , David Carmel , Adam Darlow, Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076121]
• 13: K. L. Kwok, L. Grunfeld, H. L. Sun, P. Deng. TREC 2004 Robust Track Experiments Using PIRCS. In the Online Proceedings of 2004 Text REtrieval Conference (TREC 2004).
• 14: Eric C. Jensen , Steven M. Beitzel , David Grossman , Ophir Frieder , Abdur Chowdhury, Predicting query difficulty on the web by learning visual clues, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076155]
• 15: D. Lopresti and J. Zhou, Retrieval Strategy for Noisy Text, In symposium on document analysis and information retrieval, pp 1--16, 1996.
• 16: A. Singhal, G. Salton and C. Buckley, Length normalization in degraded text collections. In symposium on document analysis and information retrieval, pp 149--162, 1996.
• 17: Elke Mittendorf, Data corruption and information retrieval, PhD Thesis, Department of Computer Science, the Katholieke Universiteit Leuven.
• 18: J. D. Gibbons and S. Chakraborty, Nonparametric statistical inference, Marcel Dekker, New York, 1992.
• 19: Bookstein, A. and Swanson,D. Probabilistic models for automatic indexing. Journal of the American Society for Information Science 25, 5(1974), 312--319.
• 20: Harter, S. P. A probabilistic approach to automatic keyword indexing. Journal of the American Society for Information Science 26, 4 and 5(1975), Part I:197--206; Part II:280--289
• 21: Malvin H. Kalos , Paula A. Whitlock, Monte Carlo methods. Vol. 1: basics, Wiley-Interscience, New York, NY, 1986
• 22: Fei Song , W. Bruce Croft, A general language model for information retrieval (poster abstract), Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.279-280, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312698]
• 23: E. Kreyszig, Advanced Enineering Mathematics, chapter 23.10, JOHN WILEY & SONS, INC. 1997.
• 24: Yun Zhou , W. Bruce Croft, Document quality models for web ad hoc retrieval, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099652]
• 25: T. Hastie, R. Tibshirani, J. H. Friedman. The Elements of Statistical Learning, Springer press, 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

General Terms:
Algorithms, Experimentation, Theory

Keywords:
query performance prediction, ranking robustness