Evaluating usability methods: why the current literature fails the practitioner

ABSTRACT

Practitioners who deal in the business world must find ways to keep up-to-date with best practices in the field and must apply them to their work in cost-effective ways. Research should help to define best practices, but often the worlds of research and of practice can seem too dichotomous. Recently, for instance, the fields of usability and user-centered design generally have seen considerable controversy about the relative effectiveness of different methodologies. In this column, Dennis Wixon argues that we need to look at whether we are evaluating methods by the appropriate criteria. He considers the growing body of literature on evaluation of methods unhelpful, or even irrelevant, to the practitioner. He argues that consideration of the factors that determine success of usability efforts in product development organizations will fundamentally change the terms of the debate. ---David A. Siegel

REFERENCES

• 1: Cook , T. D., and Campbell, D. T. Quasi-experimentation: Design and analysis issues for field settings. Rand McNally, Chicago, 1979.
• 2: Gray, W. D. , John, B.E., and Atwood, M. E. Project Ernestine: Validating a GOMS analysis for predicting and explaining real world performance. Human Computer Interaction 8, (1993), pp. 23-309.
• 3: Gray, W.D. and Saltzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human Computer Interaction 13, (1998), 203-261.
• 4: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 5: Lewis, J.R. Sample sizes for usability studies: additional considerations. Human Factors 36, (1994), pp. 368-378
• 6: Randolph G. Bias , Deborah J. Mayhew, Cost-justifying usability, Academic Press, Inc., Orlando, FL, 1994
• 7: Dennis Wixon , Judith Ramey, Field methods casebook for software design, John Wiley & Sons, Inc., New York, NY, 1996
• 8: Rudisill, M. Lewis, C. Polson, P. and McKay, T. Human Computer Interface Design: Success Cases, Emerging Methods, and Real-World Context. Morgan Kaufman, San Francisco, 1993.
• 9: Rick Spencer, The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company, Proceedings of the SIGCHI conference on Human factors in computing systems, p.353-359, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332456]
• 10: Robert A. Virzi, Refining the test phase of usability evaluation: how many subjects is enough?, Human Factors, v.34 n.4, p.457-468, Aug. 1992
• 11: Whiteside, J. Bennett, J. and Holtzblatt, K. (1988). Usability engineering: Our experience and evolution. In M. Helander (ed,) Handbook of Human Computer Interaction, (1988), North Holland, New York, pp. 791-817.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)