IR evaluation methods for retrieving highly relevant documents

ABSTRACT

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In-Query¹) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.

REFERENCES

• 1: J. Allan, J. Callan, B. Croft, L. Ballesteros, J. Broglio, J. Xu & H. Shu. INQUERY at TREC 5. In E.M. Voorhees & D.K. Harrnan (Eds.), Information technology: The Fifth Text Retrieval Conference (TREC-5). Gaithersburg, MD: National Institute of Standards and Technology, 119-132, 1997.
• 2: David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
• 3: Pia Borlund , Peter Ingwersen, Measures of relative relevance and ranked half-life: performance indicators for interactive IR, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.324-331, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291019]
• 4: W.J. Conover. Practical nonparametric statistics (2nd ed.). New York: John Wiley & Sons, 1980.
• 5: R. Green. The expression of conceptual syntagmatic relationships: A comparative survey. Journal of Documentation, 51(4): 315-338, 1995.
• 6: William R. Hersh , David H. Hickam, An evaluation of interactive Boolean and natural language searching with an online medical textbook, Journal of the American Society for Information Science, v.46 n.7, p.478-489, Aug. 1995 [doi>10.1002/(SICI)1097-4571(199508)46:7<478::AID-ASI2>3.0.CO;2-#]
• 7: P. Ingwersen & P. Willett. An introduction to algorithmic and cognitive approaches for information retrieval. Libri, 450: 160-177, 1995.
• 8: E. Michael Keen, The use of term position devices in ranked output experiments, Journal of Documentation, v.47 n.1, p.1-22, March 1991 [doi>10.1108/eb026869]
• 9: J. Kekiliinen. The effects of query complexity, expansion and structure on retrieval performance in probabilistic text retrieval. Ph.D. dissertation. Department of Information Studies, University of Tampere, 1999.
• 10: Jaana KekäläInen , Kalervo Järvelin, The Co-Effects of Query Structure and Expansion on RetrievalPerformance in Probabilistic Text Retrieval, Information Retrieval, v.1 n.4, p.329-344, January 2000 [doi>10.1023/A:1009983401464]
• 11: Jaana Kekäläinen , Kalervo Järvelin, The impact of query structure and query expansion on retrieval performance, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.130-137, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290978]
• 12: Robert M. Losee, Text retrieval and filtering: analytic models of performance, Kluwer Academic Publishers, Norwell, MA, 1998
• 13: T. B. Rajashekar , W. Bruce Croft, Combining automatic and manual index representations in probabilistic retrieval, Journal of the American Society for Information Science, v.46 n.4, p.272-283, May 1995 [doi>10.1002/(SICI)1097-4571(199505)46:4<272::AID-ASI4>3.0.CO;2-T]
• 14: S.E. Robertson & N.J. Belkin. Ranking in principle. Journal of Documentation, 34(2): 93-100, 1978.
• 15: T. Saracevic, P. Kantor, A. Chamis & D. Tnvison. A study of information seeking and retrieving. I. Background and methodology. Journal of the American Society for Information Science, 39(3): 161-176, 1988.
• 16: Steve Smithson, Information retrieval evaluation in practice: a case study approach, Information Processing and Management: an International Journal, v.30 n.2, p.205-221, March-April 1994 [doi>10.1016/0306-4573(94)90065-5]
• 17: E. Sormunen. A Method for Measuring Wtde Range PerformancIe of Boolean Queries in Full-Text Databases. Ph.D. dssertation. Department of Information Studies, University of Tampere, 2000.
• 18: Howard Robert Turtle, Inference networks for document retrieval, University of Massachusetts, Amherst, MA, 1991

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Design, Experimentation, Human Factors, Performance, Reliability