Undo and erase events as indicators of usability problems

ABSTRACT

One approach to reducing the costs of usability testing is to facilitate the automatic detection of critical incidents: serious breakdowns in interaction that stand out during software use. This research evaluates the use of undo and erase events as indicators of critical incidents in Google SketchUp (a 3D-modeling application), measuring an indicator's usefulness by the numbers and types of usability problems discovered. We compared problems identified using undo and erase events to problems identified using the user-reported critical incident technique [Hartson and Castillo 1998]. In a within-subjects experiment with 35 participants, undo and erase episodes together revealed over 90% of the problems rated as severe, several of which would not have been discovered by self-report alone. Moreover, problems found by all three methods were rated as significantly more severe than those identified by only a subset of methods. These results suggest that undo and erase events will serve as useful complements to user-reported critical incidents for low cost usability evaluation of creation-oriented applications like SketchUp.

REFERENCES

• 1: Capra, M. Contemporaneous Versus Retrospective User-Reported Critical Incidents in Usability Evaluation. In Proc. Human Factors 2002. HFES(2002), 1973--1977.
• 2: Cockton, G. and Lavery, D. A Framework for Usability Problem Extraction. In Proc. Interact 1999. IOS Press(1999), 344--352.
• 3: Cronbach, L. J. Coefficient alpha and the internal structure of tests. Psychometrika 16, 3 (1951), 297--334.
• 4: del Galdo, E.M., Williges, B.H., and Wixon, D.R. An Evaluation of Critical Incidents for Software Documentation Design. In Proc. Human Factors 1986. HFES(1986), 19--23.
• 5: Flanagan, J.C. The Critical Incident Technique. Psychological Bulletin 51, 4 (1954), 327--358.
• 6: Google SketchUp, http://sketchup.google.com
• 7: Google SketchUp New User Tutorial Videos 1-3, http://www.youtube.com/user/SketchUpVideo.
• 8: Zhiwei Guan , Shirley Lee , Elisabeth Cuddihy , Judith Ramey, The validity of the stimulated retrospective think-aloud method as measured by eye tracking, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124961]
• 9: H. Rex Hartson , José C. Castillo , John Kelso , Wayne C. Neale, Remote evaluation: the network as an extension of the usability laboratory, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.228-235, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238511]
• 10: H. Rex Hartson , José C. Castillo, Remote evaluation for post-deployment usability improvement, Proceedings of the working conference on Advanced visual interfaces, May 24-27, 1998, L'Aquila, Italy [doi>10.1145/948496.948499]
• 11: David M. Hilbert , David F. Redmiles, An approach to large-scale collection of application usage data over the Internet, Proceedings of the 20th international conference on Software engineering, p.136-145, April 19-25, 1998, Kyoto, Japan
• 12: Howarth, J., Andre, T. S., and Hartson, R. A Structured Process for Transforming Usability Data into Usability Information. Journal of Usability Studies, 3,1 (2007), 7--23.
• 13: Niels Ebbe Jacobsen , Morten Hertzum , Bonnie E. John, The evaluator effect in usability tests, CHI 98 conference summary on Human factors in computing systems, p.255-256, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286737]
• 14: Kirsh, D. and Maglio, P. On distinguishing epistemic from pragmatic action. Cog. Sci. 18 (1994), 513--549.
• 15: Jürgen Koenemann-Belliveau , John M. Carroll , Mary Beth Rosson , Mark K. Singley, Comparative usability evaluation: critical incidents and critical threads, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.245-251, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191755]
• 16: Effie Lai-Chong Law , Ebba Thora Hvannberg, Analysis of combinatorial user effect in international usability tests, Proceedings of the SIGCHI conference on Human factors in computing systems, p.9-16, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985694]
• 17: Gitte Lindgaard , Jarinee Chattratichart, Usability testing: what have we overlooked?, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240839]
• 18: Medlock, M.C., Wixon, D., Terrano, M., Romero, R., and Fulton, B. Using the RITE method to improve products: a definition and a case study. In Usability Professionals Association (2002).
• 19: Jakob Nielsen , Thomas K. Landauer, A mathematical model of the finding of usability problems, Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems, p.206-213, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169166]
• 20: Nisbett, R. and Wilson, T. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231--259, 1977.
• 21: Jeffrey Rubin , Theresa Hudson, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, Inc., New York, NY, 1994
• 22: Jared Spool , Will Schroeder, Testing web sites: five users is nowhere near enough, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634236]
• 23: Swallow, J., Hameluck, D., and Carey, T. User Interface Instrumentation for Usability Analysis: A Case Study. In Proc. Cascon 97, 1997.
• 24: Terry Winograd , Fernando Flores, Understanding computers and cognition, Ablex Publishing Corp., Norwood, NJ, 1986

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Experimentation, Human Factors

Keywords:
critical incidents, erase, google sketchup, undo, usability testing, user-reported critcial incident technique