Estimating the relative usability of two interfaces: heuristic, formal, and empirical methods compared

ABSTRACT

Two alternative user interface designs were subjected to user testing to measure user performance in a database query task. User performance was also estimated heuristically in three different ways and by use of formal GOMS modelling. The estimated values for absolute user performance had very high variability, but estimates of the relative advantage of the fastest interface were less variable. Choosing the fastest of the two designs would have a net present value more than 1,000 times the cost of getting the estimates. A software manager would make the correct choice every time in our case study if decisions were based on at least three independent estimates. User testing was 4.9 times as expensive as the cheapest heuristic method but provided better performance estimates.

REFERENCES

• 1: Victoria Bellotti, Implications of current design practice for the use of HCI techniques, Proceedings of the Fourth Conference of the British Computer Society on People and computers IV, p.13-34, October 1988, Univ. of Manchester, United Kingdom
• 2: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• 3: Wayne D. Gray , Bonnie E. John , Michael E. Atwood, The precis of Project Ernestine or an overview of a validation of GOMS, Proceedings of the SIGCHI conference on Human factors in computing systems, p.307-312, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142821]
• 4: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 5: Claire-Marie Karat , Robert Campbell , Tarra Fiegel, Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.397-404, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142873]
• 6: Robert Mack , Jakob Nielsen, Usability inspection methods: report on a workshop held at CHI'92, Monterey, CA, May 3–4, 1992, ACM SIGCHI Bulletin, v.25 n.1, p.28-33, Jan. 1993 [doi>10.1145/157203.157207]
• 7: Marilyn M. Mantei , Toby J. Teorey, Cost/benefit analysis for incorporating human factors in the software lifecycle, Communications of the ACM, v.31 n.4, p.428-439, April 1988 [doi>10.1145/42404.42408]
• 8: Jakob Nielsen, The Usability Engineering Life Cycle, Computer, v.25 n.3, p.12-22, March 1992 [doi>10.1109/2.121503]
• 9: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 10: Jakob Nielsen, Reliability of severity estimates for usability problems found by heuristic evaluation, Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems, May 03-07, 1992, Monterey, California [doi>10.1145/1125021.1125117]
• 11: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 12: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 13: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 14: Olson, J.R., and Nilsen, E. (1988). Analysis of the cognition involved in spreadsheet software interaction. Human- Computer Interaction 3, 4, 309-349.
• 15: Olson, J.R., and Olson, G.M. (1990). The growth of cognitive modeling in human-computer interaction since GOMS. Human-Computer Interaction 5, 2&3, 221-265.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Design, Human Factors, Performance

Keywords:
GOMS, absolute performance, cost-benefit estimates, heuristic estimation, heuristic evaluation, relative performance, usability, user performance, user testing