Ranking retrieval systems without relevance judgments

ABSTRACT

The most prevalent experimental methodology for comparing the effectiveness of information retrieval systems requires a test collection, composed of a set of documents, a set of query topics, and a set of relevance judgments indicating which documents are relevant to which topics. It is well known that relevance judgments are not infallible, but recent retrospective investigation into results from the Text REtrieval Conference (TREC) has shown that differences in human judgments of relevance do not affect the relative measured performance of retrieval systems. Based on this result, we propose and describe the initial results of a new evaluation methodology which replaces human relevance judgments with a randomly selected mapping of documents to topics which we refer to aspseudo-relevance judgments.Rankings of systems with our methodology correlate positively with official TREC rankings, although the performance of the top systems is not predicted well. The correlations are stable over a variety of pool depths and sampling techniques. With improvements, such a methodology could be useful in evaluating systems such as World-Wide Web search engines, where the set of documents changes too often to make traditional collection construction techniques practical.

REFERENCES

• 1: Brian T. Bartell , Garrison W. Cottrell , Richard K. Belew, Automatic combination of multiple ranked retrieval systems, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.173-181, July 03-06, 1994, Dublin, Ireland
• 2: Gordon V. Cormack , Christopher R. Palmer , Charles L. A. Clarke, Efficient construction of large test collections, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.282-289, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291009]
• 3: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, August 1998
• 4: Stephen P. Harter, Variations in relevance assessments and the measurement of retrieval effectiveness, Journal of the American Society for Information Science, v.47 n.1, p.37-49, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I]
• 5: Tefko Saracevic, Information science, Journal of the American Society for Information Science, v.50 n.12, p.1051-1063, Oct. 1999 [doi>10.1002/(SICI)1097-4571(1999)50:12<1051::AID-ASI2>3.0.CO;2-Z]
• 6: Tefko Saracevic, Evaluation of evaluation in information retrieval, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.138-146, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215351]
• 7: Amanda Spink , Howard Greisdorf, Regions and levels: measuring and mapping users' relevance judgments, Journal of the American Society for Information Science and Technology, v.52 n.2, p.161-173, Jan 2001 [doi>10.1002/1097-4571(2000)9999:9999<::AID-ASI1564>3.3.CO;2-C]
• 8: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 9: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 10: Ellen M.Voorhees and Donna Harman.Overview of the Eighth Text REtrieval Conference (TREC-8).In E.M.Voorhees and D.K.Harman,editors,The Eighth Text REtrieval Conference ,NISTSpecial Publication 500-246.National Institute of Standards and Technology,Gaithersburg,MD,November 1999.
• 11: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Retrieval models

General Terms:
Design, Experimentation, Performance