Rank-biased precision for measurement of retrieval effectiveness

ABSTRACT

A range of methods for measuring the effectiveness of information retrieval systems has been proposed. These are typically intended to provide a quantitative single-value summary of a document ranking relative to a query. However, many of these measures have failings. For example, recall is not well founded as a measure of satisfaction, since the user of an actual system cannot judge recall. Average precision is derived from recall, and suffers from the same problem. In addition, average precision lacks key stability properties that are needed for robust experiments. In this article, we introduce a new effectiveness metric, rank-biased precision, that avoids these problems. Rank-biased pre-cision is derived from a simple model of user behavior, is robust if answer rankings are extended to greater depths, and allows accurate quantification of experimental uncertainty, even when only partial relevance judgments are available.

REFERENCES

• 1: James Allan , Ben Carterette , Joshua Lewis, When will information retrieval be "good enough"?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076109]
• 2: Javed A. Aslam , Virgil Pavlu , Emine Yilmaz, A statistical method for system evaluation using incomplete judgments, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148263]
• 3: Javed A. Aslam , Emine Yilmaz , Virgiliu Pavlu, A geometric interpretation of r-precision and its correlation with average precision, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076134]
• 4: Pia Borlund , Peter Ingwersen, Measures of relative relevance and ranked half-life: performance indicators for interactive IR, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.324-331, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291019]
• 5: Chris Buckley , Ellen M. Voorhees, Retrieval evaluation with incomplete information, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009000]
• 6: Buckley, C. and Voorhees, E. M. 2005. Retrieval system evaluation. In TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge, MA, Chapter 3, 53--75.
• 7: Stefan Büttcher , Charles L. A. Clarke , Peter C. K. Yeung , Ian Soboroff, Reliable information retrieval evaluation with incomplete and biased judgements, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277755]
• 8: Ben Carterette , James Allan , Ramesh Sitaraman, Minimal test collections for retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148219]
• 9: Cooper, W. S. 1968. Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems. Amer. Document. 19, 1 (Jan.), 30--41.
• 10: Cooper, W. S. 1973. On selecting a measure of retrieval effectiveness: Part I, the ‘subjective’ philosophy of evaluation. J. Amer. Soc. Inform. Sci. 24, 87--100.
• 11: Gordon V. Cormack , Thomas R. Lynam, Statistical precision of information retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148262]
• 12: Vincenzo Della Mea , Stefano Mizzaro, Measuring retrieval effectiveness: a new proposal and a first experimental validation, Journal of the American Society for Information Science and Technology, v.55 n.6, p.530-543, April 2004 [doi>10.1002/asi.10408]
• 13: H. P. Frei , P. Schäuble, Determining the effectiveness of retrieval algorithms, Information Processing and Management: an International Journal, v.27 n.2-3, p.153-164, Feb./March 1991 [doi>10.1016/0306-4573(91)90046-O]
• 14: Donna Harman, Overview of the second text retrieval conference (TREC-2), Information Processing and Management: an International Journal, v.31 n.3, p.271-289, May-June 1995
• 15: Stephen P. Harter, Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science, v.47 n.1, p.37-49, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I]
• 16: Kartik Hosanagar, A utility theoretic approach to determining optimal wait times in distributed information retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076052]
• 17: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 18: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Geri Gay, Accurately interpreting clickthrough data as implicit feedback, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076063]
• 19: Yuri Kagolovsky , Jochen R. Moehr, Current Status of the Evaluation of Information Retrieval, Journal of Medical Systems, v.27 n.5, p.409-424, October 2003 [doi>10.1023/A:1025603704680]
• 20: E. Michael Keen, Presenting results of experimental retrieval comparisons, Information Processing and Management: an International Journal, v.28 n.4, p.491-502, July-Aug. 1992 [doi>10.1016/0306-4573(92)90006-L]
• 21: Jaana Kekäläinen, Binary and graded relevance in IR evaluations: comparison of the effects on ranking of IR systems, Information Processing and Management: an International Journal, v.41 n.5, p.1019-1033, September 2005 [doi>10.1016/j.ipm.2005.01.004]
• 22: Robert M. Losee, When information retrieval measures agree about the relative quality of document rankings, Journal of the American Society for Information Science, v.51 n.9, p.834-840, July 2000 [doi>10.1002/(SICI)1097-4571(2000)51:9<834::AID-ASI60>3.3.CO;2-T]
• 23: Marchionini, G., Moffat, A., Tate, J., Baeza-Yates, R., and Ziviani, N., Eds. 2005. Proceedings of the Twenty-Eighth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, New York, NY.
• 24: Meng, X. and Chen, Z. 2004. On user-oriented measurements of effectiveness of Web information retrieval systems. In Proceedings of the International Conference on Internet Computing, H. R. Arabnia, O. Droegehorn, and S. Chatterjee, Eds. CSREA Press, Las Vegas, NV, 527--533.
• 25: Stefano Mizzaro, Relevance: the whole history, Journal of the American Society for Information Science, v.48 n.9, p.810-832, Sept. 1997 [doi>10.1002/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U]
• 26: Alistair Moffat , William Webber , Justin Zobel, Strategic system comparisons via targeted relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277806]
• 27: Vijay Raghavan , Peter Bollmann , Gwang S. Jung, A critical investigation of recall and precision as measures of retrieval system performance, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.205-229, July 1989 [doi>10.1145/65943.65945]
• 28: Sakai, T. 2004. Ranking the NTCIR systems based on multigrade relevance. In Proceedings of the AIRS Asian Information Retrieval Symposium. Lecture Notes in Computer Science, vol. 3411. Springer, Berlin, Germany, 251--262.
• 29: Tetsuya Sakai, Alternatives to Bpref, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277756]
• 30: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 31: Tefko Saracevic, Evaluation of evaluation in information retrieval, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.138-146, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215351]
• 32: Shaw, Jr., W. M. 1986. On the foundation of evaluation. J. Amer. Soc. Inform. Sci. 37, 5, 346--348.
• 33: Louise T. Su, The relevance of recall and precision in user evaluation, Journal of the American Society for Information Science, v.45 n.3, p.207-217, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1]
• 34: Jean Tague-Sutcliffe, The pragmatics of information retrieval experimentation, revisited, Information Processing and Management: an International Journal, v.28 n.4, p.467-490, July-Aug. 1992 [doi>10.1016/0306-4573(92)90005-K]
• 35: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 36: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001
• 37: William Webber , Alistair Moffat , Justin Zobel, Score standardization for inter-collection comparison of retrieval systems, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390346]
• 38: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 39: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
Recall, average precision, pooling, precision, relevance