Relevance thresholds in system evaluations

ABSTRACT

We introduce and explore the concept of an individual's relevance threshold as a way of reconciling differences in outcomes between batch and user experiments.

REFERENCES

• 1: Azzah Al-Maskari , Mark Sanderson , Paul Clough, The relationship between IR effectiveness measures and user satisfaction, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277902]
• 2: James Allan , Ben Carterette , Joshua Lewis, When will information retrieval be "good enough"?, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076109]
• 3: C. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2004 terabyte track. Gaithersburg, MD, 2005.
• 4: G.A Gescheider. Psychophysics: method, theory and application. Lawrence Erlbaum Ass., New Jersey, 1985.
• 5: William Hersh , Andrew Turpin , Susan Price , Benjamin Chan , Dale Kramer , Lynetta Sacherek , Daniel Olson, Do batch and user evaluations give the same results?, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.17-24, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345539]
• 6: D. Kelly, X. Fu, and C. Shah. Effects of rank and precision of search results on users' evaluations of system performance. TR-2007-02, U. of North Carolina, 2007.
• 7: Andrew H. Turpin , William Hersh, Why batch and user evaluations do not give the same results, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.225-231, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383992]
• 8: Andrew Turpin , Falk Scholer, User performance versus precision measures for simple search tasks, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148176]
• 9: Ellen M. Voorhees , Donna K. Harman, TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing), The MIT Press, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.m Miscellaneous

General Terms:
Design, Experimentation, Human Factors, Performance

Keywords:
information retrieval evaluation, search engines, user study