Analytical usability evaluation for digital libraries: a case study

ABSTRACT

There are two main kinds of approach to considering usability of any system: empirical and analytical. Empirical techniques involve testing systems with users, whereas analytical techniques involve usability personnel assessing systems using established theories and methods. We report here on a set of studies in which four different techniques were applied to various digital libraries, focusing on the strengths, limitations and scope of each approach. Two of the techniques, Heuristic Evaluation and Cognitive Walkthrough, were applied in text-book fashion, because there was no obvious way to contextualize them to the Digital Libraries (DL) domain. For the third, Claims Analysis, it was possible to develop a set of re-usable scenarios and personas that relate the approach specifically to DL development. The fourth technique, CASSM, relates explicitly to the DL domain by combining empirical data with an analytical approach. We have found that Heuristic Evaluation and Cognitive Walkthrough only address superficial aspects of interface design (but are good for that), whereas Claims Analysis and CASSM can help identify deeper conceptual difficulties (but demand greater skill of the analyst). However, none fit seamlessly with existing digital library development practices, highlighting an important area for further work to support improved usability.

REFERENCES

• 1: Attfield, S., Blandford, A., & Dowell, J. Information seeking in the context of writing: a design psychology interpretation of the 'problematic situation' Journal of Documentation 59 4, 2003, 430--453.
• 2: Bates, M, J. Wilde, D. N. & Siegfried, S. Research Practices of Humanities Scholars in an Online Environment: The Getty Online Searching Project Report No 3 Library and Information Science Research 17, 1995, 5--40.
• 3: Bates, M. The Design of Browsing and Berrypicking techniques for the Online Search Interface Online Review, 13, 5, 1989, 407--424
• 4: Marcia J. Bates, The cascade of interactions in the digital library interface, Information Processing and Management: an International Journal, v.38 n.3, p.381-400, May 2002 [doi>10.1016/S0306-4573(01)00041-3]
• 5: Blandford, A, Keith, S. Lodder, T, & Fields, B Claims Analysis 'in the wild': a case study on Digital Libraries Submitted for journal publication Available from www.uclic.ucl.ac.uk/annb/DLUsability/DLtools html
• 6: Ann Blandford , George Buchanan, Workshop report: usability of digital libraries @ JCDL'02, ACM SIGIR Forum, v.36 n.2, Fall 2002 [doi>10.1145/792550.792566]
• 7: Ann Blandford , George Buchanan, Usability for digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544374]
• 8: Ann Blandford , Hanna Stelmaszewska , Nick Bryan-Kinns, Use of multiple digital libraries: a case study, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.179-188, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379479]
• 9: Christine L. Borgman, From Gutenberg to the global information infrastructure: access to information in the networked world, MIT Press, Cambridge, MA, 2000
• 10: John M. Carroll , Mary Beth Rosson, Getting around the task-artifact cycle: how to make claims and design by scenario, ACM Transactions on Information Systems (TOIS), v.10 n.2, p.181-212, April 1992 [doi>10.1145/146802.146834]
• 11: Cockton, G. Woolrych, A. Hall, L. & Hindmarch, M. Changing Analysts' Tunes: the Surprising Impact of a New Instrument for Usability Inspection Method Assessment In: People and Computers XVII: Proceedings of HCI'03, Springer, 2003, 145--161.
• 12: Connell, I. Green, T. & Blandford, A. Ontological Sketch Models: Highlighting User-System Misfits In E O'Neill, P Palanque & P. Johnson (eds.) People and Computers XVII, Proc HCI'03 Springer, 2003, 163--178.
• 13: Alan Cooper , Paul Saffo, The Inmates Are Running the Asylum, Macmillan Publishing Co., Inc., Indianapolis, IN, 1999
• 14: Elke Duncker, Cross-cultural usability of the library metaphor, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544269]
• 15: Ellis, D. & Haugan, M. Modelling the information seeking patterns of engineers and research scientists in an industrial environment Journal of Documentation 53, 4, 1997, 384--403
• 16: Hertzum, M. and Jacobsen, N. E. The Evaluator Effect: A Chilling Fact about Usability Evaluation Methods. International Journal of Human-Computer Interaction, 13.4, 2001, 421--443.
• 17: Ingwersen, P. Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory Journal of Documentation 52.1, 1996, 3--50.
• 18: Ann Blandford , George Buchanan, Usability for digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544374]
• 19: Anita Komlodi , Gary Marchionini, Key frame preview techniques for video browsing, Proceedings of the third ACM conference on Digital libraries, p.118-125, June 23-26, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276675.276688]
• 20: Kuhlthau, C. C. & Tama, S. L. Information search process of lawyers, a call for 'just for me' information services Journal of Documentation, 57.1, 2001, 25--43.
• 21: Kuhlthau, C. Longitudinal case studies of the information search process of users in libraries Library and information science research 10.3, 1988, 257--304.
• 22: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 23: Ioannis Papadakis , Ioannis Andreou , Vassileios Chrissikopoulos, Interactive Search Results, Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries, p.448-462, September 16-18, 2002
• 24: Sandusky, R. J. Digital Library Attributes: Framing usability research. In A Blandford & G Buchanan (eds.) Proc Workshop on Usability of Digital Libraries at JCDL'02 35--38, 2002 Available from www.uclic.ucl.ac.uk/annb/DLUsability/JCDL02 html http://www.uclic ucl.ac.uk/annb/DLUsability/JCDL02 html
• 25: Rick Spencer, The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company, Proceedings of the SIGCHI conference on Human factors in computing systems, p.353-359, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332456]
• 26: Stelmaszewska, H. & Blandford, A. Patterns of interactions: user behaviour in response to search results. In A Blandford & G. Buchanan (eds.) Proc JCDL Workshop on Usability 2002 Available from www.uclic.ucl.ac.uk/annb/DLUsability/JCDL02
• 27: A. G. Sutcliffe , J. M. Carroll, Designing claims for reuse in interactive systems design, International Journal of Human-Computer Studies, v.50 n.3, p.213-241, March 1999 [doi>10.1006/ijhc.1999.0245]
• 28: Vakkari, P. A. theory of the task-based information retrieval process: a summary and generalisation of a longitudinal study Journal of Documentation 57.1, 2001, 46--60.
• 29: Warren, P. R. Why they still cannot use their library catalogues Proceedings of Informing Science Conference June 19--22 2001 Krakow, Poland Cracow University of Economics 2001 Available from http://ecommerce.lebow.drexel.edu/eli/
• 30: Cathleen Wharton , John Rieman , Clayton Lewis , Peter Polson, The cognitive walkthrough method: a practitioner's guide, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: User issues

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design; Evaluation/methodology; Theory and methods

General Terms:
Design, Human Factors

Keywords:
cassm, claims analysis, cognitive walkthrough, digital libraries, heuristic evaluation, personas, scenario-based design, scenarios, usability evaluation