Let your users do the testing: a comparison of three remote asynchronous usability testing methods

ABSTRACT

Remote asynchronous usability testing is characterized by both a spatial and temporal separation of users and evaluators. This has the potential both to reduce practical problems with securing user attendance and to allow direct involvement of users in usability testing. In this paper, we report from an empirical study where we systematically compared three methods for remote asynchronous usability testing: user-reported critical incidents, forum-based online reporting and discussion, and diary-based longitudinal user reporting. In addition, conventional laboratory-based think-aloud testing was included as a benchmark for the remote methods. The results show that each remote asynchronous method supports identification of a considerable number of usability problems. Although this is only about half of the problems identified with the conventional method, it requires significantly less time. This makes remote asynchronous methods an appealing possibility for usability testing in many software projects.

REFERENCES

• 1: Morten Sieker Andreasen , Henrik Villemann Nielsen , Simon Ormholt Schrøder , Jan Stage, What happened to remote usability testing?: an empirical study of three methods, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240838]
• 2: Irene Au , Richard Boardman , Robin Jeffries , Patrick Larvie , Antonella Pavese , Jens Riegelsberger , Kerry Rodden , Molly Stevens, User experience at google: focus on the user and all else will follow, CHI '08 extended abstracts on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1358628.1358912]
• 3: Jakob Otkjær Bak , Kim Nguyen , Peter Risgaard , Jan Stage, Obstacles to usability evaluation in practice: a survey of software development organizations, Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, October 20-22, 2008, Lund, Sweden [doi>10.1145/1463160.1463164]
• 4: A.J. Bernheim Brush , Morgan Ames , Janet Davis, A comparison of synchronous remote and local usability studies for an expert interface, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986018]
• 5: Capra, M. G. An Exploration of End-User Critical Incident Classification. Master thesis, Virginia Polytechnic Institute and State University, 2001.
• 6: Castillo, J. C. The User-Reported Critical Incident Method for Remote Usability Evaluation. Master thesis, Virginia Polytechnic Institute and State University, 1997.
• 7: José C. Castillo , H. Rex Hartson , Deborah Hix, Remote usability evaluation: can users report their own critical incidents?, CHI 98 conference summary on Human factors in computing systems, p.253-254, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286736]
• 8: Heather W. Desurvire, Faster, cheaper!! Are usability inspection methods as effective as empirical testing?, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 9: Susan Dray , David Siegel, Remote possibilities?: international usability testing at a distance, interactions, v.11 n.2, March + April 2004 [doi>10.1145/971258.971264]
• 10: Ericsson, K. A. and Simon, H. A. How to study thinking in everyday life: contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity 5, 3 (1998), 178--186.
• 11: Følstad, A., Brandtzæg, P. B. and Heim, J. Usability Analysis and Evaluation of Mobile ICT Systems. http://www.hft.org/HFT01/paper01/mobility/25_01.pdf
• 12: Monty Hammontree , Paul Weiler , Nandini Nayak, Remote usability testing, interactions, v.1 n.3, p.21-25, July 1994 [doi>10.1145/182966.182969]
• 13: H. Rex Hartson , José C. Castillo, Remote evaluation for post-deployment usability improvement, Proceedings of the working conference on Advanced visual interfaces, May 24-27, 1998, L'Aquila, Italy [doi>10.1145/948496.948499]
• 14: H. Rex Hartson , José C. Castillo , John Kelso , Wayne C. Neale, Remote evaluation: the network as an extension of the usability laboratory, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.228-235, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238511]
• 15: Hertzum, M. and Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15, 1 (2003), 183--204.
• 16: David M. Hilbert , David F. Redmiles, Separating the wheat from the chaff in Internet-mediated user feedback expectation-driven event monitoring, ACM SIGGROUP Bulletin, v.20 n.1, p.35-40, Apr. 1999 [doi>10.1145/327556.327611]
• 17: Jesper Kjeldskov , Mikael B. Skov , Jan Stage, Instant data analysis: conducting usability evaluations in a day, Proceedings of the third Nordic conference on Human-computer interaction, p.233-240, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028050]
• 18: Larsen, J. M. Playful Interaction. Master thesis, Aalborg University, Department of Computer Science, 2008.
• 19: Stephanie Larissa Marsh , Jason Dykes , Fenia Attilakou, Evaluating a Geovisualization Prototype with Two Approaches: Remote Instructional vs. Face-to-Face Exploratory, Proceedings of the conference on Information Visualization, p.310-315, July 05-07, 2006 [doi>10.1109/IV.2006.51]
• 20: David R. Millen, Remote usability evaluation: user participation in the design of a Web-based email service, ACM SIGGROUP Bulletin, v.20 n.1, p.40-45, Apr. 1999 [doi>10.1145/327556.327616]
• 21: Christian Monrad Nielsen , Michael Overgaard , Michael Bach Pedersen , Jan Stage , Sigge Stenild, It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field, Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles, p.272-280, October 14-18, 2006, Oslo, Norway [doi>10.1145/1182475.1182504]
• 22: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 23: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 24: Jakob Nielsen, Usability inspection methods, Conference companion on Human factors in computing systems, p.377-378, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223730]
• 25: Olmsted, E. and Gill, M. In-person usability study compared with self-administered web (remote-different time-place) study: does mode of study produce similar results? Proceedings of UPA 2005, UPA (2005).
• 26: Helen Petrie , Fraser Hamilton , Neil King , Pete Pavan, Remote usability evaluations With disabled people, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124942]
• 27: Jeffrey Rubin , Theresa Hudson, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, Inc., New York, NY, 1994
• 28: Jean Scholtz, A Case Study: Developing a Remote, Rapid, and Automated Usability Testing Methodology for On-line Books, Proceedings of the Thirty-Second Annual Hawaii International Conference on System Sciences-Volume 2, p.2015, January 05-08, 1999
• 29: Jean Scholtz , Laura Downey, Methods for Identifying Usability Problems with Web Sites, Proceedings of the IFIP TC2/TC13 WG2.7/WG13.4 Seventh Working Conference on Engineering for Human-Computer Interaction, p.191-206, September 14-18, 1998
• 30: Michelle Potts Steves , Emile Morse , Carl Gutwin , Saul Greenberg, A comparison of usage evaluation and inspection methods for assessing groupware usability, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500306]
• 31: Thompson, J. A. Investigating the Effectiveness of Applying the Critical Incident Technique to Remote Usability Evaluation. Master thesis, Virginia Polytechnic Institute and State University, 1999.
• 32: Tullis, T., Fleischman, S., McNulty,M., Cianchette, C. and Bergel, M. An empirical comparison of lab and remote usability testing of web sites. http://home.comcast.net/~tomtullis/publications/RemoteVsLab.pdf
• 33: Vermeeren, A. P. O. S., van Kesteren, I. and Bekker, M. M. Managing the 'evaluator effect' in user testing. Proceedings of INTERACT 2003, IOS Press (2003).
• 34: Sarah Waterson , James A. Landay , Tara Matthews, In the lab and out in the wild: remote web usability testing for mobile devices, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506602]
• 35: Ryan West , Katherine Lehman, Automated summative usability studies: an empirical evaluation, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124867]
• 36: Winckler, M. A. A., Freitas, C. M. D. S. and de Lima, J. V.Remote usability testing: a case study. Proceedings of OzCHI 1999, CHISIG (1999).
• 37: Marco A. A. Winckler , Carla M. D. S. Freitas , José Valdeni de Lima, Usability remote evaluation for WWW, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633367]
• 38: --ijö, R. and Mantere, J. Are Non-Expert Usability Evaluations Valuable? http://www.hft.org/HFT01/paper01/acceptance/2_01.pdf

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

General Terms:
Experimentation

Keywords:
asynchronous testing, empirical study, remote testing, usability testing