Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?

ABSTRACT

Usability comprises the aspects effectiveness, efficiency, and satisfaction. The correlations between these aspects are not well understood for complex tasks. We present data from an experiment where 87 subjects solved 20 information retrieval tasks concerning programming problems. The correlation between efficiency, as indicated by task completion time, and effectiveness, as indicated by quality of solution, was negligible. Generally, the correlations among the usability aspects depend in a complex way on the application domain, the user's experience, and the use context. Going through three years of CHI Proceedings, we find that 11 out of 19 experimental studies involving complex tasks account for only one or two aspects of usability. When these studies make claims concerning overall usability, they rely on risky assumptions about correlations between usability aspects. Unless domain specific studies suggest otherwise, effectiveness, efficiency, and satisfaction should be considered independent aspect of usability and all be included in usability testing.

REFERENCES

• 1: Bailey, R.W. Performance vs. preference, in Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (Seattle WA, October 1993), HFES, 282-285.
• 2: Bevan, N. Measuring usability as quality of use. Software Quality Journal 4 (1995), 115-150.
• 3: Stuart K. Card , Thomas P. Moran , Allen Newell, The keystroke-level model for user performance time with interactive systems, Communications of the ACM, v.23 n.7, p.396-410, July 1980 [doi>10.1145/358886.358895]
• 4: Cohen, J. and Cohen, P. Applied Multiple Regression~Correlation Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, Hillsdale NJ, 1975.
• 5: Carl Gutwin , Saul Greenberg, Effects of awareness support on groupware usability, Proceedings of the SIGCHI conference on Human factors in computing systems, p.511-518, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274713]
• 6: Morten Hertzum , Erik Frøkjær, Browsing and querying in online documentation: a study of user interfaces and the interaction process, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.2, p.136-161, June 1996 [doi>10.1145/230562.230570]
• 7: ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability (1998).
• 8: Kirakowski, J. and Corbett, M. SUMI: The software usability measurement inventory. British Journal of Educational Technology 24, 3 (1993), 210-212.
• 9: Naur, P. Intuition in software development, in H. Ehring et al. Formal Methods and Software Development, Vol. 2, Lecture Notes in Computer Science 186, Springer, Berlin, 1985. Also in Naur, P. Computing: A Human Activity. ACM Press/Addison- Wesley, New York, 1992, 449-466.
• 10: Newman, W. and Taylor, A. Toward a methodology employing critical parameters to deliver performance improvements in interactive systems, in Proceedings of INTERACT 99 (Edinburgh, August 1999), IOS Press, 605-612.
• 11: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 12: Jakob Nielsen , Jonathan Levy, Measuring usability: preference vs. performance, Communications of the ACM, v.37 n.4, p.66-75, April 1994 [doi>10.1145/175276.175282]
• 13: Nosa Omoigui , Liwei He , Anoop Gupta , Jonathan Grudin , Elizabeth Sanocki, Time-compression: systems concerns, usage, and benefits, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.136-143, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303017]
• 14: Rasmussen, J. Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Transactions on Systems, Man, and Cybernetics SMC-13, 3 (1983), 257-266.
• 15: Louise T. Su, Evaluation measures for interactive information retrieval, Information Processing and Management: an International Journal, v.28 n.4, p.503-516, July-Aug. 1992 [doi>10.1016/0306-4573(92)90007-M]
• 16: Van Welie, M., van der Veer, G.C. and Eli~ns, A. Breaking Down Usability, in Proceedings of INTERACT 99 (Edinburgh, August 1999), IOS Press, 613-620.
• 17: Marilyn A. Walker , Jeanne Fromer , Giuseppe Di Fabbrizio , Craig Mestel , Don Hindle, What can I say?: evaluating a spoken language interface to Email, Proceedings of the SIGCHI conference on Human factors in computing systems, p.582-589, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274722]

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
  H.3 INFORMATION STORAGE AND RETRIEVAL
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Design, Experimentation, Human Factors, Management, Measurement, Performance, Theory

Keywords:
effectiveness, efficiency, information retrival, satisfaction, usability measures, usability testing, user studies