Retrievability: an evaluation measure for higher order information access tasks

ABSTRACT

Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.

REFERENCES

• 1: Leif Azzopardi , Maarten de Rijke, Automatic construction of known-item finding test beds, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148276]
• 2: L. Azzopardi and V. Vinay. Accessibility in information retrieval. In Proceedings of ECIR'08, pages 482--489, Glasgow, Scotland, 2008.
• 3: Mark Baillie , Leif Azzopardi , Ian Ruthven, Evaluating epistemic uncertainty under incomplete assessments, Information Processing and Management: an International Journal, v.44 n.2, p.811-837, March, 2008 [doi>10.1016/j.ipm.2007.04.002]
• 4: Jamie Callan , Margaret Connell, Query-based sampling of text databases, ACM Transactions on Information Systems (TOIS), v.19 n.2, p.97-130, April 2001 [doi>10.1145/382979.383040]
• 5: Nick Craswell , Stephen Robertson , Hugo Zaragoza , Michael Taylor, Relevance weighting for query independent evidence, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076106]
• 6: Anirban Dasgupta , Arpita Ghosh , Ravi Kumar , Christopher Olston , Sandeep Pandey , Andrew Tomkins, The discoverability of the web, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242630]
• 7: J. L. Gastwirth. The estimation of the lorenz curve and gini index. The Review of Economics and Statistics, 54(3):306--316, 1972.
• 8: S. Lawrence and C. L. Giles. Accessibility of information on the web. Nature, 400(6740):107--107, 1999.
• 9: Abbe Mowshowitz , Akira Kawaguchi, Assessing bias in search engines, Information Processing and Management: an International Journal, v.38 n.1, p.141-156, January 2002 [doi>10.1016/S0306-4573(01)00020-6]
• 10: Sandeep Pandey , Kedar Dhamdhere , Christopher Olston, WIC: a general-purpose algorithm for monitoring web information sources, Proceedings of the Thirtieth international conference on Very large data bases, p.360-371, August 31-September 03, 2004, Toronto, Canada
• 11: Vaclav Petricek , Tobias Escher , Ingemar J. Cox , Helen Margetts, The web structure of e-government - developing a methodology for quantitative evaluation, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135875]
• 12: Mark Sanderson , Justin Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076064]
• 13: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 14: T. Upstill, N. Craswell, and D. Hawking. Buying bestsellers online: A case study in search & searchability. In 7th Australasian Document Computing Symposium, Sydney, Australia, 2002.
• 15: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 16: Liwen Vaughan , Mike Thelwall, Search engine coverage bias: evidence and possible causes, Information Processing and Management: an International Journal, v.40 n.4, p.693-707, May 2004 [doi>10.1016/S0306-4573(03)00063-3]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Experimentation, Measurement, Performance

Keywords:
accessibililty, retrievability