A method to standardize usability metrics into a single score

ABSTRACT

Current methods to represent system or task usability in a single metric do not include all the ANSI and ISO defined usability aspects: effectiveness, efficiency & satisfaction. We propose a method to simplify all the ANSI and ISO aspects of usability into a single, standardized and summated usability metric (SUM). In four data sets, totaling 1860 task observations, we show that these aspects of usability are correlated and equally weighted and present a quantitative model for usability. Using standardization techniques from Six Sigma, we propose a scalable process for standardizing disparate usability metrics and show how Principal Components Analysis can be used to establish appropriate weighting for a summated model. SUM provides one continuous variable for summative usability evaluations that can be used in regression analysis, hypothesis testing and usability reporting.

REFERENCES

• 1: Abran, A., Surya, W., Khelifi, A., Rilling, J., Seffah, A., Robert, F. (2003). Consolidating the ISO Usability Models. Paper presented at 11th annual International Software Quality Management Conference.
• 2: ANSI (2001). Common industry format for usability test reports (ANSI-NCITS 354-2001). Washington, DC: American National Standards Institute.
• 3: Elmaoun M. Babiker , Hiroko Fujihara , Craig D. B. Boyle, A metric for hypertext usability, Proceedings of the 9th annual international conference on Systems documentation, p.95-104, October 1991, Chicago, Illinois, United States [doi>10.1145/122778.122793]
• 4: Breyfogle, F. (1999). Implementing Six Sigma: Smarter Solutions Using Statistical Methods. John Wiley and Sons.
• 5: Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P. Jordan, B. Thomas, and B. Weerdmeester (Eds.), Usability Evaluation in Industry (pp.189--194). London: Taylor and Francis. See also http://www.cee.hw.ac.uk/~ph/sus.html
• 6: Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245--276.
• 7: John P. Chin , Virginia A. Diehl , Kent L. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-218, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57203]
• 8: Cordes, R. E (1984). Application of Magnitude Estimation for Evaluating Software Ease of Use. In Gavriel Salvendy (Ed.) First USA-Japan Conference on Human Computer Interaction, Amsterdam: Elsevier Science Publishers.
• 9: Joseph S. Dumas, User-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 10: Dunteman, George H, (1989) Principal Components Analysis. In Sage University Papers Series Quantitative Applications in the Social Sciences; No. 07-069 Newbury Park Sage Publications, Inc.
• 11: Joseph S. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Intellect Books, Exeter, UK, 1999
• 12: Erik Frøkjær , Morten Hertzum , Kasper Hornbæk, Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.345-352, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332455]
• 13: Gliem, J. and Gliem, R. (2003). Calculating, Interpreting, and Reporting Cronbach's Alpha Reliability Coefficient for Likert-Type Scales. In 2003 Midwest Research to Practice Conference in Adult, Continuing and Community Education. Columbus, OH.
• 14: Joseph F. Hair, Jr. , Rolph E. Anderson , Ronald L. Tatham , William C. Black, Multivariate data analysis (4th ed.): with readings, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995
• 15: Hagen, E. (Ed.) (1976). Human Reliability Analysis, control and instrumentation. Nuclear Safety. 17(3), 315--326.
• 16: Marc Hassenzahl , Nina Sandweg, From mental effort to perceived usability: transforming experiences into summary assessments, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986044]
• 17: Harry, M. J (1987). The Nature of Six Sigma Quality. Technical Report, Government Electronics Group, Motorola Inc. Scottsdale, AZ.
• 18: ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva, Switzerland: Author.
• 19: Jolliffe, I. T. (1972). Discarding variables in a principal component analysis 1: Artificial data. Applied Statistics, 21, 160--173.
• 20: Jolliffe, Ian T.(2002). Principal Component Analysis. Secaucus, NJ, USA: Springer-Verlag.
• 21: Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational Psychology Measurement, 20, 141--151.
• 22: Kirakowski, J. (1996). The Software Usability Measurement Inventory: Background and usage. In P. Jordan, B. Thomas, and B. Weerdmeester (Eds.), Usability Evaluation in Industry (pp. 169--178). London, UK: Taylor and Francis. (Also, see http://www.ucc.ie/hfrg/questionnaires/sumi/index.html).
• 23: Kirakowski, J., and Corbett, M. (1993). SUMI: The Software Usability Measurement Inventory. British Journal of Educational Technology, 24, 210--212.
• 24: Lewis, J. R (1982) "Testing Small System Customer Setup" in Proceedings of the Human Factors Society 26th Annual Meeting p. 718--720.
• 25: Lewis, J (1991) A Rank-Based Method for the Usability Comparison of Competing Products. In Proceedings of the Human Factors and Ergonomics Society 35th Annual Meeting San Francisco California (pp1312--1316).
• 26: James R. Lewis, Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ, ACM SIGCHI Bulletin, v.23 n.1, p.78-81, Jan. 1991 [doi>10.1145/122672.122692]
• 27: Lewis, J. R. (1992). Psychometric evaluation of the Post-Study System Usability Questionnaire: The PSSUQ. In Proceedings of the Human Factors Society 36th Annual Meeting (pp. 1259--1263). Atlanta, GA: Human Factors Society.
• 28: Lewis, J. R. (1993). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use (Tech. Report 54.786). Boca Raton, FL: IBM Corp. http://drjim.0catch.com/usabqtr.pdf
• 29: James R. Lewis, IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use, International Journal of Human-Computer Interaction, v.7 n.1, p.57-78, Jan.-March 1995 [doi>10.1080/10447319509526110]
• 30: Lewis, J.R. (In Press). Handbook of Human Factors and Ergonomics 3rd Edition. Gavriel Salvendy Editor. John Wiley & Sons.
• 31: Martin,P and Bateson, P (1993) Measuring Behaviour. (2nd Edition) Cambridge {England}; New York, NY, USA: Cambridge University Press.
• 32: McGee, M. (2003). Usability magnitude estimation. Proc. HFES, 47th Annual Meeting, (691--695).
• 33: Mick McGee, Master usability scaling: magnitude estimation and master scaling applied to usability measurement, Proceedings of the SIGCHI conference on Human factors in computing systems, p.335-342, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985735]
• 34: McIver, J. P., & Carmines, E. G. (1981). Unidimensional scaling. Thousand Oaks, CA: Sage.
• 35: Rolf Molich , Meghan R. Ede , Klaus Kaasgaard , Barbara Karyukin, Comparative usability evaluation, Behaviour & Information Technology, v.23 n.1, p.65-74, January-February 2004 [doi>10.1080/0144929032000173951]
• 36: Morrison, D. F. (1976). Multivariate statistical methods (2nd ed.). New York: McGraw-Hill.
• 37: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 38: Jakob Nielsen , Jonathan Levy, Measuring usability: preference vs. performance, Communications of the ACM, v.37 n.4, p.66-75, April 1994 [doi>10.1145/175276.175282]
• 39: Nielsen, J. (1997). Usability testing. In G. Salvendy (Ed.), The Handbook of Human Factors and Ergonomics, (2rd Edition). John Wiley & Sons.
• 40: Nunnally, J. C. (1978). Psychometric Theory. New York, NY: McGraw-Hill.
• 41: Park, Kyung S. (1997). Human Error. In Gavriel Salvendy (Ed.), The Handbook of Human Factors and Ergonomics, (2rd Edition). John Wiley & Sons.
• 42: Sauro, J. (2004) How long should a task take? Identifying Spec Limits for Task Times in Usability Tests. Retrieved September 13, 2004, from Measuring Usability Web site : http://measuringusability.com/time_specs.htm
• 43: Sauro, J. (2004) How Do You Calculate a Z-Score? Retrieved September 13, 2004, from Measuring Usability Web site: http://measuringusability.com/z_calc.htm
• 44: Sauro, J & Kindlund E. (In Press) Making Sense of Usability Metrics: Usability and Six Sigma, in Proceedings of the 14th Annual Conference of the Usability Professionals Association, Montreal, Canada.
• 45: Stevens, S.S. (1975). Psychophysics: Introduction to its Perceptual, Neural, and Social Prospects. New York: John Wiley.
• 46: Whiteside, J., Bennett, J. and Holtzblatt, K. (1988) "Usability Engineering: Our Experience and Evolution" in The Handbook of Human Computer Interaction Elsevier Science Publishers, Amsterdam, pp 791--817.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Standardization

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Benchmarking; Evaluation/methodology

Keywords:
measurement, principal components analysis, six sigma, standardization, usability