Applying user testing data to UEM performance metrics

ABSTRACT

The lack of standard assessment criteria for reliably comparing usability evaluation methods (UEMs) is an important gap in HCI knowledge. Recently, metrics for assessing thoroughness, validity, and effectiveness of UEMs, based on user data, have been proposed to bridge this gap. This paper reports our findings of applying these proposed metrics in a study that compared heuristic evaluation (HE) to HE-Plus (an extended version of HE). Our experiment showed better overlap among the HE-Plus evaluators than the HE evaluators, demonstrating greater reliability of the method. When evaluation data, from testing the usability of the same website, was used in calculating the UEM performance metrics, HE-Plus was found to be a superior method to HE in all assessment criteria with a 17%, 39%, and 67% improvement in the aspects of thoroughness, validity, and effectiveness, respectively. The paper concludes with a discussion concerning the limitations of the effectiveness of the UEM from which the real users' data was obtained.

REFERENCES

• 1: Chattratichart, J. and Brodie, J. Extending the heuristic evaluation method through contextualisation. Proc. HFES2002, HFES (2002), 641--645.
• 2: Gilbert Cockton , Alan Woolrych, Sale must end: should discount methods be cleared off HCI's shelves?, interactions, v.9 n.5, September 2002 [doi>10.1145/566981.566990]
• 3: Gray, W. D. and Salzman, M. C. Damaged Merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13 (1998), 203--261.
• 4: Hartson, H. R., Andre, T. S. and Williges, R. C. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15, 1 (2003), 145--181.
• 5: Hertzum, M. and Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15, 1 (2003), 183--204.
• 6: Martin Kessner , Jo Wood , Richard F. Dillon , Robert L. West, On the reliability of usability testing, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634127]
• 7: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Performance, Reliability

Keywords:
heuristic evaluation