Do batch and user evaluations give the same results?

ABSTRACT

Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

REFERENCES

• 1: John P. Chin , Virginia A. Diehl , Kent L. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-218, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57203]
• 2: C. Cleverdon and E. Keen, Factors determining the performance of indexin systems, Cranfield UK: Aslib Cranfield Research Project 1966.
• 3: Donna Harman, Overview of the first TREC conference, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.36-47, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160692]
• 4: William Hersh, Relevance and retrieval evaluation: perspectives from medicine, Journal of the American Society for Information Science, v.45 n.3, p.201-206, April 1994 [doi>10.1002/(SICI)1097-4571(199404)45:3<201::AID-ASI9>3.0.CO;2-W]
• 5: Eric Lagergren , Paul Over, Comparing interactive information retrieval systems across sites: the TREC-6 interactive track matrix experiment, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.164-172, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290986]
• 6: C. Meadow, Relevance?, Journal of the American Society for Information Science, 36: 354-355, 1985.
• 7: S. E. Robertson , S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.232-241, July 03-06, 1994, Dublin, Ireland
• 8: Amit Singhal , Chris Buckley , Mandar Mitra, Pivoted document length normalization, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-29, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243206]
• 9: Karen Sparck Jones, Information Retrieval Experiment, Butterworth-Heinemann, Newton, MA, 1981
• 10: D. Swanson, Information retrieval as a trial-anderror process, Library Quarterly, 47: 128-148, 1977.
• 11: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 12: Justin Zobel , Alistair Moffat, Exploring the similarity space, ACM SIGIR Forum, v.32 n.1, p.18-34, Spring 1998 [doi>10.1145/281250.281256]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Experimentation, Human Factors, Performance