Statistical precision of information retrieval evaluation

ABSTRACT

We introduce and validate bootstrap techniques to compute confidence intervals that quantify the effect of test-collection variability on average precision (AP) and mean average precision (MAP) IR effectiveness measures. We consider the test collection in IR evaluation to be a representative of a population of materially similar collections, whose documents are drawn from an infinite pool with similar characteristics. Our model accurately predicts the degree of concordance between system results on randomly selected halves of the TREC-6 ad hoc corpus. We advance a framework for statistical evaluation that uses the same general framework to model other sources of chance variation as a source of input for meta-analysis techniques.

REFERENCES

• 1: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 2: Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291009]
• 3: Efron, B., and Tsibirani, R. J. An Introduction to the Bootstrap. Chapman and Hall, New York, 1994.
• 4: Fisher, R. A. Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society 22 (1925), 700--725.
• 5: Glass, G. V. Meta-analysis at 25. http://glass.ed.asu.edu/gene/papers/meta25.html, 2000.
• 6: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 7: Lenhard, J. Models and statistical inference: The controversy between Fisher and Neyman-Pearson. British Journal for the Philosophy of Science (2006).
• 8: Rothman, K. J., and Greenland, S. Modern Epidemiology. Lippincott Williams & Wilkins, 1998.
• 9: Mark Sanderson , Hideo Joho, Forming test collections with no system pooling, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009001]
• 10: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 11: Jacques Savoy, Statistical inference in retrieval effectiveness evaluation, Information Processing and Management: an International Journal, v.33 n.4, p.495-512, July 1997 [doi>10.1016/S0306-4573(97)00027-7]
• 12: Jean Tague-Sutcliffe, The pragmatics of information retrieval experimentation, revisited, Information Processing and Management: an International Journal, v.28 n.4, p.467-490, July-Aug. 1992 [doi>10.1016/0306-4573(92)90005-K]
• 13: Tague-Sutcliffe, J., and Blustein, J. A statistical analysis of the TREC-3 data. In Proceedings of TREC-3, The Third Information Retrieval Conference (1994), pp. 385--398.
• 14: Voorhees, E., and Harman, D. Overview of the Sixth Text REtrieval Conference (TREC-6). In 6th Text REtrieval Conference (Gaithersburg, MD, 1997).
• 15: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 16: Voorhees, E. M. Overview of the TREC-2004 robust track. In 13th Text REtrieval Conference (Gaithersburg, MD, 2004).
• 17: Ellen M. Voorhees , Chris Buckley, The effect of topic set size on retrieval experiment error, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564432]
• 18: Chris Buckley , Ellen M. Voorhees, Retrieval evaluation with incomplete information, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009000]
• 19: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005
• 20: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
bootstrap, confidence interval, precision