Evaluation over thousands of queries

ABSTRACT

Information retrieval evaluation has typically been performed over several dozen queries, each judged to near-completeness. There has been a great deal of recent work on evaluation over much smaller judgment sets: how to select the best set of documents to judge and how to estimate evaluation measures when few judgments are available. In light of this, it should be possible to evaluate over many more queries without much more total judging effort. The Million Query Track at TREC 2007 used two document selection algorithms to acquire relevance judgments for more than 1,800 queries. We present results of the track, along with deeper analysis: investigating tradeoffs between the number of queries and number of judgments shows that, up to a point, evaluation over more queries with fewer judgments is more cost-effective and as reliable as fewer queries with more judgments. Total assessor effort can be reduced by 95% with no appreciable increase in evaluation errors.

REFERENCES

• 1: J. Allan, B. Carterette, J. A. Aslam, V. Pavlu, B. Dachev, and E. Kanoulas. Overview of the TREC 2007 Million Query Track. In Proceedings of TREC, 2007.
• 2: J. A. Aslam and V. Pavlu. A practical sampling strategy for efficient retrieval evaluation, technical report.
• 3: J. A. Aslam and V. Pavlu. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of ECIR, pages 198--209. 2007.
• 4: Javed A. Aslam , Virgiliu Pavlu , Emine Yilmaz, Measure-based metasearch, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076133]
• 5: Javed A. Aslam , Virgil Pavlu , Emine Yilmaz, A statistical method for system evaluation using incomplete judgments, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148263]
• 6: David Bodoff , Pu Li, Test theory for assessing IR test collections, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277805]
• 7: R. L. Brennan. Generalizability Theory. Springer-Verlag, New York, 2001.
• 8: K. R. W. Brewer and M. Hanif. Sampling With Unequal Probabilities. Springer, New York, 1983..
• 9: Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148284]
• 10: Ben Carterette, Robust test collections for retrieval evaluation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277754]
• 11: Ben Carterette , James Allan , Ramesh Sitaraman, Minimal test collections for retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148219]
• 12: Ben Carterette , Mark D. Smucker, Hypothesis testing with incomplete relevance judgments, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321530]
• 13: C. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2004 Terabyte Track. In Proceedings of TREC, 2004.
• 14: C. L. A. Clarke, F. Scholer, and I. Soboroff. The TREC 2005 terabyte track. In Proceedings of TREC, 2005.
• 15: Eric C. Jensen , Ophir Frieder, Repeatable evaluation of information retrieval effectiveness in dynamic environments, Illinois Institute of Technology, Chicago, IL, 2006
• 16: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 17: K. Sparck Jones and C. J. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59--75, 1976.
• 18: W. L. Stevens. Sampling without replacement with probability proportional to size. Journal of the Royal Statistical Society. Series B (Methodological), Vol. 20, No. 2. (1958), pp. 393--397
• 19: S. K. Thompson. Sampling. Wiley Series in Probability and Mathematical Statistics, 1992.
• 20: Emine Yilmaz , Javed A. Aslam, Estimating average precision with incomplete and imperfect judgments, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183633]
• 21: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
evaluation, information retrieval, million query track, test collections