On the reliability of usability testing

ABSTRACT

Six professional usability testing teams conducted a usability test on an early prototype of a dialog box. Altogether, they identified 36 usability problems. No problem was detected by every team, 2 were found by five teams, 4 by four teams, 7 by three teams, 7 by two teams, and 18 problems were identified by one team only. There was more agreement among teams in this study compared to a previous study [1] and there was more agreement among the teams on severe vs. minor problems. Implications for the cooperation between usability testers and their clients are discussed.

REFERENCES

• 1: Rolf Molich , Ann Damgaard Thomsen , Barbara Karyukina , Lars Schmidt , Meghan Ede , Wilma van Oel , Meeta Arcuri, Comparative evaluation of usability tests, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632768]
• 2: Molich, R. Comparative Usability Evaluation -- CUE. Available at http://www.dialogdesign.dk/cue.html.
• 3: Spencer, R., Miner, T., Gilman, E., Fraser, K., Story, L., & Palmen, H. Observed sources of variability in usability reports. Manuscript submitted for publication, 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Training, help, and documentation

General Terms:
Design, Human Factors, Reliability

Keywords:
problem severity, reliability, usability testing