Some measures and procedures for evaluation of the user interface in an information retrieval system

ABSTRACT

Planning the evaluation of an information retrieval system involves two steps: first, a determination of performance descriptors and measures appropriate to the system objectives and, secondly, a development of an evaluation design which ensures the effect of variation in components of interest will be isolated and assessed in an unbiased fashion. This paper examines the question of retrieval system evaluation from the perspective of the user. It presents evaluation procedures which are appropriate to this perspective and which can be used to isolate the effect of variation in the user interface to the system. The general procedure is exemplified by an application to evaluation of an experimental OPAC interface.

REFERENCES

• BELKIN82: Belkin N.J., Oddy R.N., Brooks H.M., "ASK for Information Retrieval. Part 1. Background and Theory", Journal of Documentation 38(2), 1982, pp. 61-71.
• DERVIN84: Dervin B., "The Information Needs of Californians- 1984; Report #2- Context, Summary, Conclusions, Implications, Applications", Institute of Governmental Affairs, University of California, Davis, 1984.
• FARRADANE73: Farradane J., "On the Inverse Relationship of Recall and Precision," Journal of Documentation 29(2), 1973, pp. 229-30.
• MCCAIN87: Katherine W. McCain , Howard D. White , Belver C. Griffith, Comparing retrieval performance in online data bases, Information Processing and Management: an International Journal, v.23 n.6, p.539-553, Sept. 2, 1987 [doi>10.1016/0306-4573(87)90058-6]
• ROBERTS79: Roberts F.S., "Measurement Theory", Addison-Wesley, 1979.
• SALTON83: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• STEVENS83: Stevens G.C., "User-Friendly Systems: A Critical Examination of the Concept," Behavior and Information Technology 2(1), 1983, pp. 3-16.
• TAGUE87: J. Tague, Generating an individualized user interface, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.57-60, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42013]
• TAGUE87: J. Tague, Informativeness as an ordinal utility function for information retrieval, ACM SIGIR Forum, v.21 n.3-4, p.10-17, March 1987 [doi>10.1145/30075.30077]
• WERSIG85: Wersig G., Windel G., "Information Science Needs a Theory of 'Information Actions'," Social Science Information Studies 5, 1985, pp. 11-23.
• WINOGRAD86: Terry Winograd , Fernando Flores, Understanding computers and cognition, Ablex Publishing Corp., Norwood, NJ, 1986

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

General Terms:
Design, Human Factors, Performance