Reliability of severity estimates for usability problems found by heuristic evaluation

ABSTRACT

Ratings from single evaluators are very unreliable when usability specialists judge the severity of usability problems found by heuristic evaluation, but the mean severity rating from four evaluators gets within half a rating point of the true severity 95% of the time. Also, the evaluators do agree that usability problems found by heuristic evaluation are all real problems even though each rater had originally only identified a small proportion of the problems.

REFERENCES

• 1: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 2: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]

INDEX TERMS

Keywords:
evaluation methods, judgments, prioritizing, severity, usability problems