Hypothesis testing with incomplete relevance judgments

ABSTRACT

Information retrieval experimentation generally proceeds in a cycle of development, evaluation, and hypothesis testing. Ideally, the evaluation and testing phases should be short and easy, so as to maximize the amount of time spent in development. There has been recent work on reducing the amount of assessor effort needed to evaluate retrieval systems, but it has not, for the most part, investigated the effects of these methods on tests of significance. In this work, we explore in detail the effects of reduced sets of judgments on the sign test. We demonstrate both analytically and empirically the relationship between the power of the test, the number of topics evaluated, and the number of judgments available. Using these relationships, we can determine the number of topics and judgments needed for the least-cost but highest-confidence significance evaluation. Specifically, testing pairwise significance over 192 topics with fewer than 5 judgments for each is as good as testing significance over 25 topics with an average of 166 judgments for each - 85% less effort producing no additional errors.

REFERENCES

• 1: Ben Carterette , James Allan , Ramesh Sitaraman, Minimal test collections for retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148219]
• 2: J. Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Earlbaum Associates, 1988.
• 3: Gordon V. Cormack , Thomas R. Lynam, Power and bias of subset pooling strategies, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277934]
• 4: Julian J. Faraway, Extending the Linear Model with R (Texts in Statistical Science), Chapman & Hall/CRC, 2005
• 5: J. M. Hoenig and D. M. Heisey. The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1):11--6, 2001.
• 6: Eric C. Jensen , Ophir Frieder, Repeatable evaluation of information retrieval effectiveness in dynamic environments, Illinois Institute of Technology, Chicago, IL, 2006
• 7: E. L. Lehmann. Testing Statistical Hypotheses. Springer, 1997.
• 8: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 9: Mark D. Smucker , James Allan , Ben Carterette, A comparison of statistical significance tests for information retrieval evaluation, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321528]
• 10: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 11: W. Venables and B. D. Ripley. Modern Applied Statistics with S. Springer, 2003.
• 12: E. M. Voorhees. Overview of the 2004 trec robust track. In 13th TREC, 2004.
• 13: D. Wackerly, W. Mendenhall, and R. L. Sheaffer. Mathematical Statistics With Applications. P W S Publishers, 5th edition, 1995.
• 14: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.0 General

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
evaluation, hypothesis testing, information retrieval, test collections