Usability evaluation considered harmful (some of the time)

ABSTRACT

Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done 'by rule' rather than 'by thought'. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design's scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today's reluctant reactions by users will forestall tomorrow's eager acceptance. The choice of evaluation methodology - if any - must arise from and be appropriate for the actual problem or research question under consideration.

REFERENCES

• 1: Barkhuus, L., Rode, J. From Mice to Men - 24 Years of Evaluation in CHI. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/ (2007).
• 2: Bush, V. As We May Think. Atlantic Monthly, (1945)
• 3: Buxton, B. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann, (2007).
• 4: Buxton, W. & Sniderman, R. Iteration in the Design of the Human-Computer Interface. Proc 13th. Meeting, Human Factors Assoc. of Canada, (1980), 72--81.
• 5: Christensen, C. The Innovator's Dilemma. Harper Business School Press, (1997).
• 6: Cockton, G. Make Evaluation Poverty History. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/. (2007)
• 7: Alan Dix , Janet E. Finlay , Gregory D. Abowd , Russell Beale, Human-Computer Interaction (3rd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2003
• 8: Edsger W. Dijkstra, Letters to the editor: go to statement considered harmful, Communications of the ACM, v.11 n.3, p.147-148, March 1968 [doi>10.1145/362929.362947]
• 9: W. Keith Edwards , Rebecca E. Grinter, At Home with Ubiquitous Computing: Seven Challenges, Proceedings of the 3rd international conference on Ubiquitous Computing, p.256-272, September 30-October 02, 2001, Atlanta, Georgia, USA
• 10: Engelbart, D.C. and English, W.K. A Research Center for Augmenting Human Intellect. AFIPS Fall Joint Computer Conference, Vol. 33, (1968), 395--410.
• 11: Bill Gaver , Tony Dunne , Elena Pacenti, Design: Cultural probes, interactions, v.6 n.1, p.21-29, Jan./Feb. 1999 [doi>10.1145/291224.291235]
• 12: John D. Gould, How to design usable systems, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 13: Saul Greenberg, Teaching human computer interaction to programmers, interactions, v.3 n.4, p.62-76, July/Aug. 1996 [doi>10.1145/234813.234820]
• 14: Greenberg, S. 2008. Embedding a design studio course in a conventional Computer Science program. P. Kotzé. W. Wong, J. Jorge, A. Dix and P. Silva (eds): Creativity and HCI: From Experience to Design in Education -- Selected Contributions from HCIEd 2007.
• 15: Greenberg, S. and Thimbleby, H. The weak science of human-computer interaction. Proc CHI '92 Research Symposium on HCI (1992).
• 16: Carl Gutwin , Saul Greenberg, The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces, Proceedings of the 9th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, p.98-103, June 04-16, 2000
• 17: Hewett, Baecker, Card, Carey, Gasen, Mantei, Perlman, Strong and Verplank (1996) ACM SIGCHI Curricula for Human--Computer Interaction. Last updated 2004-06-03. http://sigchi.org/cdg/index.html
• 18: Kaye, J. and Sengers, P. The Evolution of Evaluation. ACM CHI'07 - Alt.CHI. http://www.viktoria.se/altchi/
• 19: Thomas K. Landauer, Trouble with Computers: Usefulness, Usability, and Productivity, MIT Press, Cambridge, MA, 1996
• 20: Lieberman, H. The Tyranny of Evaluation. web.media.mit.edu/~lieber/Misc/TyrannyEvaluation.html, ACM CHI Fringe, (2003).
• 21: Newman, W. CHI Guide to a Successful Archive Submission. http://www.chi2008.org/archiveGuide.html, (2008).
• 22: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 23: Dan R. Olsen, Jr., Evaluating user interface systems research, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294256]
• 24: David Pinelle , Carl Gutwin, A Review of Groupware Evaluations, Proceedings of the 9th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, p.86-91, June 04-16, 2000
• 25: Snodgrass, A. & Coyne, R. Interpretation in architecture: Design as a way of thinking. London: Routledge. (2006).
• 26: R. Stanley Dicks, Mis-usability: on the uses and misuses of usability testing, Proceedings of the 20th annual international conference on Computer documentation, p.26-30, October 20-23, 2002, Toronto, Ontario, Canada [doi>10.1145/584955.584960]
• 27: Sutherland, I. Sketchpad: A man-machine graphical communication system. PhD Thesis, MIT, (1963).
• 28: Masaki Suwa , Barbara Tversky, External Representations Contribute to the Dynamic Construction of Ideas, Proceedings of the Second International Conference on Diagrammatic Representation and Inference, p.341-343, April 18-20, 2002
• 29: Harold Thimbleby, User interface design, ACM, New York, NY, 1990
• 30: Maryam Tohidi , William Buxton , Ronald Baecker , Abigail Sellen, User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback, Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles, p.105-114, October 14-18, 2006, Oslo, Norway [doi>10.1145/1182475.1182487]
• 31: Maryam Tohidi , William Buxton , Ronald Baecker , Abigail Sellen, Getting the right design and the design right, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124960]
• 32: Weightman, G. (2003) Signor Marconi's Magic Box. Da Capo Press.
• 33: Zhai, S. Evaluation is the worst form of HCI research except all those other forms that have been tried, essay published at CHI Place, (2003). http://www.almaden.ibm.com/u/zhai/publications.html

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors

Keywords:
interface critiques, teaching usability, usability testing