Meta-analysis of correlations among usability measures

ABSTRACT

Understanding the relation between usability measures seems crucial to deepen our conception of usability and to select the right measures for usability studies. We present a meta-analysis of correlations among usability measures calculated from the raw data of 73 studies. Correlations are generally low: effectiveness measures (e.g., errors) and efficiency measures (e.g., time) have a correlation of .247 ± .059 (Pearson's product-moment correlation with 95% confidence interval), efficiency and satisfaction (e.g., preference) one of .196 ± .064, and effectiveness and satisfaction one of .164 ± .062. Changes in task complexity do not influence these correlations, but use of more complex measures attenuates them. Standard questionnaires for measuring satisfaction appear more reliable than homegrown ones. Measures of users' perceptions of phenomena are generally not correlated with objective measures of the phenomena. Implications for how to measure usability are drawn and common models of usability are criticized.

REFERENCES

• 1: Arnold, H. J. & Feldman, D. C. Social Desirability Response Bias in Self-Report Choice Situations, Academy of Management Journal, 24 (1981), 377--385.
• 2: Bailey, R. Performance vs. Preference, Proc. HFES 1993, 282--286.
• 3: Bevan, N. Measuring Usability As Quality of Use, Software Quality Journal, 4 (1995), 115--150.
• 4: John P. Chin , Virginia A. Diehl , Kent L. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-218, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57203]
• 5: Cohen, J. Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates, 1969.
• 6: Cronbach, L. J. Coefficient Alpha and the Internal Structure of Tests, Psychometrika, 16 (1951), 297--333.
• 7: Czerwinski, M., Horvitz, E., & Cutrell, E. Subjective Duration Assessment: An Implicit Probe for Software Usability? Proc. IHM-HCI 2001, (2001), 167--170.
• 8: Sarah A. Douglas , Arthur E. Kirkpatrick , I. Scott MacKenzie, Testing pointing device performance and user assessment with the ISO 9241, Part 9 standard, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.215-222, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303042]
• 9: Erik Frøkjær , Morten Hertzum , Kasper Hornbæk, Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.345-352, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332455]
• 10: Glass, G., McGaw, B., & Lee Smith, M. Meta-Analysis in Social Research, Sage Publications, 1981.
• 11: Hart S. & Staveland, L., Development of NASA-TLX, in Hancock & Meshkati Human Mental Workload, Elsevier, 1988, 139--183.
• 12: Hassenzahl, M. & Tractinsky, N. User Experience -- a Research Agenda, Behaviour & Information Technology, 25, 2 (2006), 91--99.
• 13: Kasper Hornbík, Current practice in measuring usability: Challenges to usability studies and research, International Journal of Human-Computer Studies, v.64 n.2, p.79-102, February 2006 [doi>10.1016/j.ijhcs.2005.06.002]
• 14: Hunter, J. & Schmidt, F. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, Sage Publications, Thousand Oaks, CA, 2004.
• 15: ISO Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs)--Part 11: Guidance on Usability, 1998.
• 16: Jeff Sauro , Erika Kindlund, A method to standardize usability metrics into a single score, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055028]
• 17: George V. Kissel, The effect of computer experience on subjective and objective software usability measures, Conference companion on Human factors in computing systems, p.284-285, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223676]
• 18: James R. Lewis, IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use, International Journal of Human-Computer Interaction, v.7 n.1, p.57-78, Jan.-March 1995 [doi>10.1080/10447319509526110]
• 19: Lipsey, M. & Wilson, D. Practical Meta-Analysis, Sage Publications, Thousand Oaks, CA, 2000.
• 20: Mick McGee, Master usability scaling: magnitude estimation and master scaling applied to usability measurement, Proceedings of the SIGCHI conference on Human factors in computing systems, p.335-342, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985735]
• 21: McGee, M., Rich, A., & Dumas, J. Understanding the Usability Construct: User-Perceived Usability, Proc. HFES'2004, 907--911.
• 22: Newman, W. & Taylor, A. Towards a Methodology Employing Critical Parameters to Deliver Performance Improvements in Interactive Systems, Proc. INTERACT'99, 605--612.
• 23: Jakob Nielsen , Jonathan Levy, Measuring usability: preference vs. performance, Communications of the ACM, v.37 n.4, p.66-75, April 1994 [doi>10.1145/175276.175282]
• 24: Rasmussen, J. Skills, Rules, and Knowledge: Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models, IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 3 (1983), 257--266.
• 25: Rosenthal, R. Meta-Analytic Procedures for Social Research, Sage Publications, Newbury Park, CA, 1991.
• 26: Ahmed Seffah , Mohammad Donyaee , Rex B. Kline , Harkirat K. Padda, Usability measurement and metrics: A consolidated model, Software Quality Control, v.14 n.2, p.159-178, June 2006 [doi>10.1007/s11219-006-7600-8]
• 27: Brian Shackel, Usability—context, framework, definition, design and evaluation, Human factors for informatics usability, Cambridge University Press, New York, NY, 1991
• 28: Ben Shneiderman , Catherine Plaisant, Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition), Pearson Addison Wesley, 2004
• 29: Shirley Taylor , Peter Todd, Assessing IT usage: the role of prior experience, MIS Quarterly, v.19 n.4, p.561-570, Dec. 1995 [doi>10.2307/249633]
• 30: Suzanne Weisband , Sara Kiesler, Self disclosure on computer forms: meta-analysis and implications, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.3-10, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238387]
• 31: Yeh, Y.-Y. & Wickens, C. D. Dissociation of Performance and Subjective Measures of Workload, Human Factors, 30, 1 (1988), 111--120.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
ISO 9241, meta analysis, subjective satisfaction, usability evaluation, usability measures