Liberal relevance criteria of TREC -: counting on negligible documents?

ABSTRACT

Most test collections (like TREC and CLEF) for experimental research in information retrieval apply binary relevance assessments. This paper introduces a four-point relevance scale and reports the findings of a project in which TREC-7 and TREC-8 document pools on 38 topics were reassessed. The goal of the reassessment was to build a subcollection of TREC for experiments on highly relevant documents and to learn about the assessment process as well as the characteristics of a multigraded relevance corpus.Relevance criteria were defined so that a distinction was made between documents rich in topical information (relevant and highly relevant documents) and poor in topical information (marginally relevant documents). It turned out that about 50% of documents assessed as relevant were regarded as marginal. The characteristics of the relevance corpus and lessons learned from the reassessment project are discussed. The need to develop more elaborated relevance assessment schemes is emphasized.

REFERENCES

• 1: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 2: Robert Burgin, Variations in relevance judgments and the evaluation of retrieval performance, Information Processing and Management: an International Journal, v.28 n.5, p.619-627, 1992 [doi>10.1016/0306-4573(92)90031-T]
• 3: Michael Gordon , Praveen Pathak, Finding information on the World Wide Web: the retrieval effectiveness of search engines, Information Processing and Management: an International Journal, v.35 n.2, p.141-180, March 1999 [doi>10.1016/S0306-4573(98)00041-7]
• 4: Hawking, D. Overview of the TREC-9 Web Track. <http://trec.nist.gov/pubs/trec9/papers/web9.pdf> {Cited 2 January 2002}.
• 5: Kalervo Järvelin , Jaana Kekäläinen, IR evaluation methods for retrieving highly relevant documents, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.41-48, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345545]
• 6: Saracevic, T., Kantor. P. et al. A study of information seeking and retrieving. I Background and methodology. Journal of the American Society for Information Science 39, 3, 161-176.
• 7: Sormunen, E. A method for measuring wide range performance of Boolean queries in full-text databases. Doctoral Thesis. University of Tampere. Acta Electronica Universitatis Tamperensis 34, URL: <http://acta.uta.fi/teos.phtml?3786>, Tampere 2000.
• 8: Sormunen, E., Kekäläinen, J., Koivisto, J. & Jäärvelin, K. Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. Journal of Documentation 57, 3, 358--374.
• 9: TREC homepage, Data - English relevance judgements. Available at: <http://trec.nist.gov/data/reljudge_eng.html> {Cited 31 December 2001}.
• 10: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
• 11: Ellen M. Voorhees, Evaluation by highly relevant documents, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.74-82, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383963]
• 12: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 13: Voorhees, E. & Harman, D. Overview of the Seventh Text Retrieval Conference (TREC-7). 1999. http://trec.nist.gov/pubs/trec7/papers/overview_7.pdf.gz {Cited 2 January 2002}.
• 14: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation

Keywords:
TREC, experiment, graded relevance, relevance assessments, test collections