Usability testing: what have we overlooked?

ABSTRACT

For more than a decade, the number of usability test participants has been a major theme of debate among usability practitioners and researchers keen to improve usability test performance. This paper provides evidence suggesting that the focus be shifted to task coverage instead. Our data analysis of nine commercial usability test teams participating in the CUE-4 study revealed no significant correlation between the percentage of problems found or of new problems and number of test users, but correlations of both variables and number of user tasks used by each usability team were significant. The role of participant recruitment on usability test performance and future research directions are discussed.

REFERENCES

• 1: Chattratichart, J. & Jordan, P. W. Simulating 'lived' user experience -- Virtual immersion and inclusive design. In Proceedings of Interact 2003, Amsterdam: IOS Press (2003), 721--725.
• 2: Cockton, G & Woolrych, A. Understanding inspection methods: Lessons from an assessment of heuristic evaluation. In People & Computers XV, A. Blandford & J. Vanderdonckt (Eds.), Springer-Verlag (2001), 171--191.
• 3: Constantine, L. CHI 2003 Feature: Testing& 1 2 3 4 5 & Testing& http://usabilitynews.com/news/article1058.asp. May, 2003.
• 4: Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments & Computers, 35, 3, Psychonomic Society (2003), 379--383.
• 5: Fleiss, J. L. Statistical Methods for Rates and Proportions, 2nd Ed., John Wiley & Sons (1981).
• 6: Hertzum, M. & Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15, 1, Lawrence Erlbaum Associates (2003), 183--204.
• 7: Jacobsen, N. E., Hertzum, M., & John, B. E. The evaluator effect in usability studies: Problem detection and severity judgements. In Proc. HFES 1998, HFES (1998), 1336--1340.
• 8: Rolf Molich , Meghan R. Ede , Klaus Kaasgaard , Barbara Karyukin, Comparative usability evaluation, Behaviour & Information Technology, v.23 n.1, p.65-74, January-February 2004 [doi>10.1080/0144929032000173951]
• 9: Rolf Molich , Robin Jeffries, Comparative expert reviews, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766148]
• 10: Rolf Molich , Meghan R. Ede , Klaus Kaasgaard , Barbara Karyukin, Comparative usability evaluation, Behaviour & Information Technology, v.23 n.1, p.65-74, January-February 2004 [doi>10.1080/0144929032000173951]
• 11: Molich, R., Bevan, N., Curson, I., Butler, S., Kindlund, E., Miller, D., & Kirakowski, J. Comparative evaluation of usability tests. In Proc. UPA 1998, UPA (1998), 189--200.
• 12: Rolf Molich , Ann Damgaard Thomsen , Barbara Karyukina , Lars Schmidt , Meghan Ede , Wilma van Oel , Meeta Arcuri, Comparative evaluation of usability tests, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632768]
• 13: Nielsen, J. Why you only need to test with 5 users, Jakob Nielsen's Alertbox, March 19, 2000, http://www.useit.com/alertbox/20000319.html.
• 14: Jakob Nielsen , Thomas K. Landauer, A mathematical model of the finding of usability problems, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.206-213, May 1993, Amsterdam, The Netherlands
• 15: Jared Spool , Will Schroeder, Testing web sites: five users is nowhere near enough, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634236]
• 16: Robert A. Virzi, Refining the test phase of usability evaluation: how many subjects is enough?, Human Factors, v.34 n.4, p.457-468, Aug. 1992

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors, Performance

Keywords:
UEM, metrics, participant recruitment, usability testing