Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

The goal of system evaluation in information retrieval has always been to determine which of a set of systems is superior on a given collection. The tool used to determine system ordering is an evaluation metric such as average precision, which computes relative, collection-specific scores. We argue that a broader goal is achievable. In this paper we demonstrate that, by use of standardization, scores can be substantially independent of a particular collection, allowing systems to be compared even when they have been tested on different collections. Compared to current methods, our techniques provide richer information about system performance, improved clarity in outcome reporting, and greater simplicity in reviewing results from disparate sources.

REFERENCES

• 1: Azzah Al-Maskari , Mark Sanderson , Paul Clough, The relationship between IR effectiveness measures and user satisfaction, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277902]
• 2: Javed A. Aslam , Emine Yilmaz , Virgiliu Pavlu, The maximum entropy method for analyzing retrieval measures, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076042]
• 3: Javed A. Aslam , Virgil Pavlu , Emine Yilmaz, A statistical method for system evaluation using incomplete judgments, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148263]
• 4: David Bodoff , Pu Li, Test theory for assessing IR test collections, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277805]
• 5: C. Buckley. The SMART project at TREC. In Voorhees and Harman {2005}, chapter 13.
• 6: C. Buckley and E. Voorhees. Retrieval system evaluation. In Voorhees and Harman {2005}, chapter 3.
• 7: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 8: Charles L. A. Clarke , Eugene Agichtein , Susan Dumais , Ryen W. White, The influence of caption features on clickthrough patterns in web search, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277767]
• 9: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 2006
• 10: W. L. Hays. Statistics. Harcourt Brace, Fort Worth, 4th edition, 1991.
• 11: Scott B. Huffman , Michael Hochster, How well does result relevance predict session satisfaction?, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277839]
• 12: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 13: G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors. Proc. 28th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Salvador, Brazil, August 2005.
• 14: Stefano Mizzaro , Stephen Robertson, Hits hits TREC: exploring IR evaluation results with network analysis, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277824]
• 15: A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., to appear.
• 16: Tetsuya Sakai, Evaluating evaluation metrics based on the bootstrap, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148261]
• 17: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 18: Jacques Savoy, Statistical inference in retrieval effectiveness evaluation, Information Processing and Management: an International Journal, v.33 n.4, p.495-512, July 1997 [doi>10.1016/S0306-4573(97)00027-7]
• 19: Mark D. Smucker , James Allan , Ben Carterette, A comparison of statistical significance tests for information retrieval evaluation, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321528]
• 20: J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In D. K. Harman, editor, Proc. TREC-3, pages 385--398, November 1994. NIST Special Publication 500-225.
• 21: Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148176]
• 22: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005
• 23: W. Webber, A. Moffat, and J. Zobel. Score standardization for robust comparison of retrieval systems. In M.Wu, A. Turpin, and A. Spink, editors, Proc. 12th Australasian Document Computing Symposium, pages 1--8, Melbourne, December 2007.
• 24: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement, Performance

Keywords:
average precision, evaluation, retrieval experiment, system measurement