Coming to terms: comparing and combining the results of multiple evaluators performing heuristic evaluation

ABSTRACT

In this paper we describe a new way to perform heuristic evaluations, which allows multiple evaluators to easily compare and combine the results of their reviews. This method was developed to provide a single, reliable, result to the client, but it also allowed us to easily negotiate differences in our findings, and to prioritize usability problems identified by the evaluation. An unexpected side effect is that, by using this evaluation method, the practitioner can measure and predict the effect of usability improvements.

REFERENCES

• 1: Gilbert Cockton , Darryn Lavery , Alan Woolrych, Inspection-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 2: Joseph S. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Intellect Books, Exeter, UK, 1999
• 3: Morten Hertzum , Niels Ebbe Jacobsen , Rolf Molich, Usability inspections by groups of specialists: perceived agreement in spite of disparate observations, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506534]
• 4: Jacobsen, N. E., Hertzum, M., & John B. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments, Proceedings of the Human Factors and Ergonomics Society (pp. 1336--1340).
• 5: Rolf Molich , Jakob Nielsen, Improving a human-computer dialogue, Communications of the ACM, v.33 n.3, p.338-348, March 1990 [doi>10.1145/77481.77486]
• 6: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 7: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 8: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 9: Jakob Nielsen , Thomas K. Landauer, A mathematical model of the finding of usability problems, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.206-213, May 1993, Amsterdam, The Netherlands
• 10: Rubin, J. (1994). Handbook of usability testing (pp. 277--280). Portland: John Wiley and Sons.
• 11: Stanton, N., & Baber, C. (1996), Factors affecting the selection of methods and techniques prior to conducting a usability evaluation, in Jordan, P. W., Thomas, B., Weerdmeester, B. A, & McClelland, I. L. (Eds.) Usability evaluation in industry, (pp. 39--48), Bristol, PA: Taylor & Francis.
• 12: Wilson, C. (1999). Reader's questions: Severity scales. Usability Interface, Volume 5, Number 4.
• 13: Chauncey Wilson , Kara Pernice Coyne, The whiteboard: Tracking usability issues: to bug or not to bug?, interactions, v.8 n.3, p.15-19, May/June 2001 [doi>10.1145/369825.369828]
• 14: Yahoo UI Design Library, Retrieved October 19th, 2006 from http://developer.yahoo.com/ypatterns/pattern.php?pattern=ratinganobject

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors, Measurement

Keywords:
competitive analysis, evaluator effect, heuristic evaluation, rating scale, severity, usability, usability evaluation, usability inspection