A comparative evaluation of heuristic-based usability inspection methods

ABSTRACT

Given that heuristic evaluation (HE) is a popular evaluation method among practitioners despite criticisms surrounding its performance and reliability, there is a need to improve the method's performance. Several studies have shown HE-Plus, an emerging variant of HE, to outperform HE in both effectiveness and reliability. HE-Plus uses the same set of heuristics as HE; the only difference between these two methods is the 'usability problems profile' element in HE-Plus. This paper reports our attempt to verify the original profile employed in HE-Plus based on usability problem classification in the User Action Framework and an experiment evaluating its outcome by comparing HE with two HE variants using a profile (HE-Plus and HE++) and a control group. Our results confirmed the role of the 'usability problems profiles' on improving the performance and reliability of heuristic evaluation: both HE-Plus and HE++ outperformed HE in terms of effectiveness as well as reliability.

REFERENCES

• 1: AARP Audience-Centered Heuristics: Older Adults. http://www.redish.net/content/handouts/Audience-Centered_Heuristics.pdf.
• 2: Terence S. Andre , H. Rex Hartson , Steven M. Belz , Faith A. McCreary, The user action framework: a reliable foundation for usability engineering support tools, International Journal of Human-Computer Studies, v.54 n.1, p.107-136, Jan. 2001 [doi>10.1006/ijhc.2000.0441]
• 3: Bailey, R. W. Heuristic evaluation vs User testing. UI design update newsletter - January 2001, http://www.humanfactors.com/downloads/jan01.asp
• 4: Chattratichart, J. & Brodie, J. Extending the heuristic evaluation method through contextualisation. In Proceedings of the 46th Annual Meeting of the Human Factors and Ergonomics Society, HFES (2002), 641--645.
• 5: Chattratichart, J. & Brodie, J. HE-Plus -- Towards usage-centered expert review for website design. In Proc. forUse 2003, MA:Ampersand Press (2003), 155--169.
• 6: Jarinee Chattratichart , Jaqueline Brodie, Applying user testing data to UEM performance metrics, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986003]
• 7: Gray, W. D., & Salzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods, iHuman-Computer Interaction, 13 (1998), 203--262.
• 8: Hartson, H. R., Andre, T. S., & Williges, R. W. Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 15(1) (2003), 145--181.
• 9: Nina Khalayli , Silja Nyhus , Kari Hamnes , Tone Terum, Persona based rapid usability kick-off, CHI '07 extended abstracts on Human factors in computing systems, April 28-May 03, 2007, San Jose, CA, USA [doi>10.1145/1240866.1240898]
• 10: Michael D. Levi , Frederick G. Conrad, A heuristic evaluation of a World Wide Web prototype, interactions, v.3 n.4, p.50-61, July/Aug. 1996 [doi>10.1145/234813.234819]
• 11: Gitte Lindgaard , Jarinee Chattratichart, Usability testing: what have we overlooked?, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240839]
• 12: Arnold M. Lund , Lynn Strother , Wendy A. Rogers, The human factors and ergonomics society perspective, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056826]
• 13: Rolf Molich , Meghan R. Ede , Klaus Kaasgaard , Barbara Karyukin, Comparative usability evaluation, Behaviour & Information Technology, v.23 n.1, p.65-74, January-February 2004 [doi>10.1080/0144929032000173951]
• 14: Nielsen, J. Heuristic Evaluation. http://www.useit.com/papers/heuristic/.
• 15: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 16: Norman, D. A. Cognitive engineering. In D. A. Norman & S. W. Draper (Eds.) User Centered System Design: New Perspectives on Human-Computer Interaction, pp. 31--61. Hillsdale, NJ: Lawrence Erlbaum Associates (1986).
• 17: Perfetti, C. Usability Testing Best Practices: An Interview with Rolf Molich. http://www.webpronews.com/topnews/2003/07/30/usability-testing-best-practices-an-interview-with-rolf-molich.
• 18: Redish, G., Chisnell, D., & Lee, A. A new take on heuristic evaluation: Bringing personas, tasks, and heuristics together with a new model for understanding older adults as users. http://www.redish.net/content/talks.html.
• 19: Schaffer, E. Why "how many users" is just the wrong question, UI Design Newsletter -- May 2007: Insights from Human Factors International. http://www.humanfactors.com/downloads/may07.asp.
• 20: Sears, A. Heuristic Walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9(3), (1997), 213--234.
• 21: The Webby Awards Judging Criteria. http://www.webbyawards.com/entries/criteria.php.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors, Performance, Reliability

Keywords:
cue 4, heuristic evaluation, performance metrics, uem