Retrieval evaluation with incomplete information

ABSTRACT

This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the collection). We show that current evaluation measures are not robust to substantially incomplete relevance judgments. A new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets. This finding suggests that substantially larger or dynamic test collections built using current pooling practices should be viable laboratory tools, despite the fact that the relevance information will be incomplete and imperfect.

REFERENCES

• 1: Chris Buckley. trec_eval IR evaluation package. Available from ftp://ftp.cs.cornell.edu/pub/smart.
• 2: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 3: Cyril W. Cleverdon, The significance of the Cranfield tests on index languages, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122861]
• 4: Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291009]
• 5: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, August 1998
• 6: Michael B. Eisenberg, Measuring relevance judgements, Information Processing and Management: an International Journal, v.24 n.4, p.373-390, May 1988 [doi>10.1016/0306-4573(88)90042-8]
• 7: H. P. Frei , P. Schäuble, Determining the effectiveness of retrieval algorithms, Information Processing and Management: an International Journal, v.27 n.2-3, p.153-164, Feb./March 1991 [doi>10.1016/0306-4573(91)90046-O]
• 8: Google. Benefits of a Google search. http://www.google.com/technology/whyuse.html, January 2004.
• 9: Stefano Mizzaro. A new measure of retrieval effectiveness(Or: What's wrong with precision and recall). In Proceedings of the International Workshop on Information Retrieval(IR'2001), pages 43--52, 2001.
• 10: Rabia Nuray , Fazli Can, Automatic ranking of retrieval systems in imperfect environments, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860510]
• 11: Mark E. Rorvig. The simple scalability of documents. Journal of the American Society for Information Science, 41(8):590--598, 1990.
• 12: Ian Soboroff , Charles Nicholas , Patrick Cahan, Ranking retrieval systems without relevance judgments, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.66-73, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383961]
• 13: K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.
• 14: C. J. van Rijsbergen. Evaluation, chapter 7. Butterworths, 2 edition, 1979.
• 15: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 16: Ellen M. Voorhees, Evaluation by highly relevant documents, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.74-82, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383963]
• 17: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001
• 18: Ellen M. Voorhees , Chris Buckley, The effect of topic set size on retrieval experiment error, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564432]
• 19: Ellen M. Voorhees and Donna Harman. Overview of the seventh Text REtrieval Conference(TREC-7). In Proceedings of the Seventh Text REtrieval Conference(TREC-7), pages 1--23, 1999. NIST Special Publication 500--242.
• 20: Y. Y. Yao, Measuring retrieval effectiveness based on user preference of documents, Journal of the American Society for Information Science, v.46 n.2, p.133-145, March 1995 [doi>10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z]
• 21: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Experimentation, Measurement

Keywords:
cranfield, incomplete judgments