A case for improved evaluation of query difficulty prediction

ABSTRACT

Query difficulty prediction aims to identify, in advance, how well an information retrieval system will perform when faced with a particular search request. The current standard evaluation methodology involves calculating a correlation coefficient, to indicate how strongly the predicted query difficulty is related with an actual system performance measure, usually Average Precision. We run a series of experiments based on predictors that have been shown to perform well in the literature, comparing these across different TREC runs. Our results demonstrate that the current evaluation methodology is severely limited. Although it can be used to demonstrate the performance of a predictor for a single system, such performance is not consistent over a variety of retrieval systems. We conclude that published results in the query difficulty area are generally not comparable, and recommend that prediction be evaluated against a spectrum of underlying search systems.

REFERENCES

• 1: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 2: Claudia Hauff , Djoerd Hiemstra , Franciska de Jong, A survey of pre-retrieval query performance predictors, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA [doi>10.1145/1458082.1458311]
• 3: Ben He , Iadh Ounis, Query performance prediction, Information Systems, v.31 n.7, p.585-594, November 2006 [doi>10.1016/j.is.2005.11.003]
• 4: Falk Scholer , Hugh E. Williams , Andrew Turpin, Query association surrogates for Web search: Research Articles, Journal of the American Society for Information Science and Technology, v.55 n.7, p.637-650, May 2004 [doi>10.1002/asi.v55:7]
• 5: David J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, Chapman & Hall/CRC, 2007
• 6: Y. Zhao, F. Scholer, and Y. Tsegay. Effective pre-retrieval query performance prediction using similarity and variability evidence. In Proc. ECIR, pages 28--39, Glasgow, UK, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
evaluation, query difficulty prediction