Evaluating evaluation metrics based on the bootstrap

ABSTRACT

This paper describes how the Bootstrap approach to statistics can be applied to the evaluation of IR effectiveness metrics. First, we argue that Bootstrap Hypothesis Tests deserve more attention from the IR community, as they are based on fewer assumptions than traditional statistical significance tests. We then describe straightforward methods for comparing the sensitivity of IR metrics based on Bootstrap Hypothesis Tests. Unlike the heuristics-based "swap" method proposed by Voorhees and Buckley, our method estimates the performance difference required to achieve a given significance level directly from Bootstrap Hypothesis Test results. In addition, we describe a simple way of examining the accuracy of rank correlation between two metrics based on the Bootstrap Estimate of Standard Error. We demonstrate the usefulness of our methods using test collections and runs from the NTCIR CLIR track for comparing seven IR metrics, including those that can handle graded relevance and those based on the Geometric Mean.

REFERENCES

• 1: Asakawa, S. and Selberg, E.: The New MSN Search Engine Developed by Microsoft (in Japanese), Information Processing Society of Japan Magazine, Vol 46, No. 9, pp. 1008--1015, 2005.
• 2: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 3: Efron, B. and Tibshirani, R.: An Introduction to the Bootstrap, Chapman & Hall/CRC, 1993.
• 4: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 5: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 6: Johnson, D. H.: The Insignificance of Statistical Significance Testing, Journal of Wildlife Management Vol. 63, Issue. 3, pp. 763--772, 1999.
• 7: Jaana Kekäläinen, Binary and graded relevance in IR evaluations: comparison of the effects on ranking of IR systems, Information Processing and Management: an International Journal, v.41 n.5, p.1019-1033, September 2005 [doi>10.1016/j.ipm.2005.01.004]
• 8: Wei-Hao Lin , Alexander Hauptmann, Revisiting the effect of topic set size on retrieval error, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076166]
• 9: NTCIR: http://research.nii.ac.jp/ntcir/
• 10: Sakai, T.: The Effect of Topic Sampling on Sensitivity Comparisons of Information Retrieval Metrics, NTCIR-5 Proceedings, pp. 505--512, 2005.
• 11: Sakai, T. et al.: Toshiba BRIDJE at NTCIR-5 CLIR: Evaluation using Geometric Means, NTCIR-5 Proceedings, pp. 56--63, 2005.
• 12: Sakai, T.: On the Reliability of Information Retrieval Metrics based on Graded Relevance, Information Processing and Management, to appear, 2006.
• 13: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 14: Jacques Savoy, Statistical inference in retrieval effectiveness evaluation, Information Processing and Management: an International Journal, v.33 n.4, p.495-512, July 1997 [doi>10.1016/S0306-4573(97)00027-7]
• 15: Ellen M. Voorhees , Chris Buckley, The effect of topic set size on retrieval experiment error, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564432]
• 16: Voorhees, E. M.: Overview of the TREC 2004 Robust Retrieval Track, TREC 2004 Proceedings, 2005.
• 17: Vu, H.-T. and Gallinari, P.: On Effectiveness Measures and Relevance Functions in Ranking INEX Systems, AIRS 2005 Proceedings, LNCS 3689, pp. 312--327, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation

Keywords:
bootstrap, evaluation, graded relevance, test collection