Usability inspections by groups of specialists: perceived agreement in spite of disparate observations

ABSTRACT

Evaluators who examine the same system using the same usability evaluation method tend to report substantially different sets of problems. This so-called evaluator effect means that different evaluations point to considerably different revisions of the evaluated system. The first step in coping with the evaluator effect is to acknowledge its existence. In this study 11 usability specialists individually inspected a website and then met in four groups to combine their findings into group outputs. Although the overlap in reported problems between any two evaluators averaged only 9%, the 11 evaluators felt that they were largely in agreement. The evaluators perceived their disparate observations as mulitiple sources of evidence in support of the same issues, not as disagreements. Thus, the group work increased the evaluators' confidence in their individual inspections, rather than alerted them to the evaluator effect.

REFERENCES

• 1: Heath, C., and Gonzalez, R. Interaction with others increases decision confidence but not decision quality: Evidence against information collection views of interactive decision making. Organizational Behavior and Human Decision Processes 61, 3 (1995), 305--326.
• 2: Hertzum, M., and Jacobsen, N.E. The evaluator effect: A chilling fact about usability evaluation methods. To appear in International Journal of Human-Computer Interaction.
• 3: Lewis, C., and Wharton, C. Cognitive walkthroughs, in M. Helander, T.K. Landauer, and P. Prabhu (eds.), Handbook of Human-Computer Interaction. Second, completely revised edition (pp. 717--732). Elsevier, Amsterdam, 1997.
• 4: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]

INDEX TERMS

Keywords:
evaluator effect, inspections, usability evaluation methods