Testing web sites: five users is nowhere near enough

ABSTRACT

We observed the same task executed by 49 users on four production web sites. We tracked the rates of discovery of new usability problems on each site and, using that data, estimated the total number of usability problems on each site and the number of tests we would need to discover every problem. Our findings differ sharply from rules-of-thumb derived from earlier work by Virzi[1] and Nielsen[2,3] commonly viewed as "industry standards." We found that the four sites we studied would need considerably more than five users to find 85% of the problems

REFERENCES

• 1: Robert A. Virzi, Refining the test phase of usability evaluation: how many subjects is enough?, Human Factors, v.34 n.4, p.457-468, Aug. 1992
• 2: Jakob Nielsen , Thomas K. Landauer, A mathematical model of the finding of usability problems, Proceedings of the SIGCHI conference on Human factors in computing systems, p.206-213, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169166]
• 3: Nielsen, Jakob Why You Only Need To Test With Five Users, Jakob Nielsen's Alertbox, March 19, 2000, http://www.useit.com/alertbox/20000319.html

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

          Nouns: World Wide Web (WWW)
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Information browsers

General Terms:
Algorithms, Design, Performance

Keywords:
number of users, usability engineering, usability testing, web usability