Relevance judgments between TREC and Non-TREC assessors

ABSTRACT

This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents.

REFERENCES

• 1: Kalervo Järvelin , Jaana Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.41-48, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345545]
• 2: Eero Sormunen, Liberal relevance criteria of TREC -: counting on negligible documents?, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564433]
• 3: Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148176]
• 4: Pertti Vakkari , Eero Sormunen, The influence of relevance levels on the effectiveness of interactive information retrieval, Journal of the American Society for Information Science and Technology, v.55 n.11, p.963-969, September 2004 [doi>10.1002/asi.20046]
• 5: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.0 General

General Terms:
Human Factors, Measurement

Keywords:
TREC relevance assessment, user study