Comparative analysis of high- and low-fidelity prototypes for more valid usability evaluations of mobile devices

ABSTRACT

Validation of low-fidelity prototyping test results is difficult because we cannot claim whether the results are the effect of the prototype itself or the essence of the design concept we try to evaluate. However, it will cost too much if we implement a fully functional prototype for more valid evaluation. In this research, we provide a qualitative and reflective analysis of usability evaluations of a text messaging functionality of a mobile phone by comparing three types of prototyping techniques---paper-based and computer-based and fully functional prototype. This analysis led us to realize how significantly the unique characteristics of each different prototype affect the usability evaluation in different ways. We identify what characteristics of each prototype causes the differences in finding usability problems, and then suggest key considerations for designing more valid low-fidelity prototypes based on this analysis.

REFERENCES

• 1: N. P. Archer , Y. Yuan, Comparing telephone-computer interface designs: are software simulations as good as hardware prototypes?, International Journal of Human-Computer Studies, v.42 n.2, p.169-184, Feb. 1995 [doi>10.1006/ijhc.1995.1008]
• 2: Daniel Avrahami , Scott E. Hudson, Forming interactivity: a tool for rapid prototyping of physical interactive products, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, June 25-28, 2002, London, England [doi>10.1145/778712.778735]
• 3: Marion Buchenau , Jane Fulton Suri, Experience prototyping, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.424-433, August 17-19, 2000, New York City, New York, United States [doi>10.1145/347642.347802]
• 4: Gilbert Cockton , Alan Woolrych, Sale must end: should discount methods be cleared off HCI's shelves?, interactions, v.9 n.5, September 2002 [doi>10.1145/566981.566990]
• 5: Gregorio Convertino , Dennis C. Neale , Laurian Hobby , John M. Carroll , Mary Beth Rosson, A laboratory method for studying activity awareness, Proceedings of the third Nordic conference on Human-computer interaction, p.313-322, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028063]
• 6: Nigel Davies , James Landay , Scott Hudson , Albrecht Schmidt, Guest Editors' Introduction: Rapid Prototyping for Ubiquitous Computing, IEEE Pervasive Computing, v.4 n.4, p.15-17, October 2005 [doi>10.1109/MPRV.2005.78]
• 7: Helen M. Grady, Web site design: a case study in usability testing using paper prototypes, Proceedings of IEEE professional communication society international professional communication conference and Proceedings of the 18th annual ACM international conference on Computer documentation: technology & teamwork, September 24-27, 2000, Cambridge, Massachusetts
• 8: Saul Greenberg , Michael Boyle, Customizable physical interfaces for interacting with conventional applications, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.571991]
• 9: O. Gutierrez, Prototyping techniques for different problem contexts, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.259-264, March 1989
• 10: Bonnie E. John , Dario D. Salvucci, Multipurpose Prototypes for Assessing User Interfaces in Pervasive Computing Systems, IEEE Pervasive Computing, v.4 n.4, p.27-34, October 2005 [doi>10.1109/MPRV.2005.80]
• 11: Johnny C. Lee , Daniel Avrahami , Scott E. Hudson , Jodi Forlizzi , Paul H. Dietz , Darren Leigh, The calder toolkit: wired and wireless components for rapidly prototyping interactive devices, Proceedings of the 2004 conference on Designing interactive systems: processes, practices, methods, and techniques, August 01-04, 2004, Cambridge, MA, USA [doi>10.1145/1013115.1013139]
• 12: Linchuan Liu , Peter Khooshabeh, Paper or interactive?: a study of prototyping techniques for ubiquitous computing environments, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766132]
• 13: Blair MacIntyre , Maribeth Gandy , Steven Dow , Jay David Bolter, DART: a toolkit for rapid design exploration of augmented reality experiences, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029669]
• 14: Tek-Jin Nam , Woohun Lee, Integrating hardware and software: augmented reality based prototyping method for digital products, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766092]
• 15: Jakob Nielsen , Thomas K. Landauer, A mathematical model of the finding of usability problems, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.206-213, May 1993, Amsterdam, The Netherlands
• 16: Celine Pering, Interaction design prototyping of communicator devices: towards meeting the hardware-software challenge, interactions, v.9 n.6, p.36-46, November + December 2002 [doi>10.1145/581951.581952]
• 17: Derek Reilly , Dave Dearman , Michael Welsman-Dinelle , Kori Inkpen, Evaluating Early Prototypes in Context: Trade-offs, Challenges, and Successes, IEEE Pervasive Computing, v.4 n.4, p.42-50, October 2005 [doi>10.1109/MPRV.2005.76]
• 18: Marc Rettig, Prototyping for tiny fingers, Communications of the ACM, v.37 n.4, p.21-27, April 1994 [doi>10.1145/175276.175288]
• 19: Jim Rudd , Ken Stern , Scott Isensee, Low vs. high-fidelity prototyping debate, interactions, v.3 n.1, p.76-85, Jan. 1996 [doi>10.1145/223500.223514]
• 20: Kurt Schneider, Prototypes as assets, not toys: why and how to extract knowledge from prototypes, Proceedings of the 18th international conference on Software engineering, p.522-531, March 25-29, 1996, Berlin, Germany
• 21: Reinhard Sefelin , Manfred Tscheligi , Verena Giller, Paper prototyping - what is it good for?: a comparison of paper- and computer-based low-fidelity prototyping, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765986]
• 22: Snyder, C. Paper Prototyping: The Fast and Easy Way to Define and Refine User Interfaces. Morgan Kaufmann Publishers, San Francisco, CA, 2003.
• 23: Michael Thompson , Nina Wishbow, Prototyping: tools and techniques: improving software and documentation quality through rapid prototyping, Proceedings of the 10th annual international conference on Systems documentation, p.191-199, October 13-16, 1992, Ottawa, Ontario, Canada [doi>10.1145/147001.147030]
• 24: Robert A. Virzi , Jeffrey L. Sokolov , Demetrios Karis, Usability problem identification using both low- and high-fidelity prototypes, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.236-243, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238516]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques

Keywords:
designing mobile devices, prototyping, usability evaluation