Comparison of empirical testing and walkthrough methods in user interface evaluation

ABSTRACT

We investigated the relative effectiveness of empirical usability testing and individual and team walkthrough methods in identifying usability problems in two graphical user interface office systems. The findings were replicated across the two systems and show that the empirical testing condition identified the largest number of problems, and identified a significant number of relatively severe problems that were missed by the walkthrough conditions. Team walkthroughs achieved better results than individual walkthroughs in some areas. About a third of the significant usability problems identified were common across all methods. Cost-effectiveness data show that empirical testing required the same or less time to identify each problem when compared to walkthroughs.

REFERENCES

• 1: IEEE Software Staff, Interface-Walkthroughs: Efficient Collaborative Testing, IEEE Software, v.8 n.5, p.94-95, September 1991 [doi>10.1109/52.84220]
• 2: Victoria Bellotti, Implications of current design practice for the use of HCI techniques, Proceedings of the Fourth Conference of the British Computer Society on People and computers IV, p.13-34, October 1988, Univ. of Manchester, United Kingdom
• 3: Carroll, J., Smith-Kerker, P.L., Ford, J.R., and Mazur- Rimetz, S.A. The minimal manual. Human-Computer Interaction, 3, 1987-88, pp. 123-153.
• 4: Heather Desurvire , Debbie Lawrence , Michael Atwood, EMPIRICISM VERSUS JUDGEMENT: COMPARING USER INTERFACE EVALUATION METHODS ON A NEW TELEPHONE-BASED INTERFACE, ACM SIGCHI Bulletin, v.23 n.4, p.58-59, Oct. 1991 [doi>10.1145/126729.1056062]
• 5: John D. Gould , Clayton Lewis, Designing for usability: key principles and what designers think, Communications of the ACM, v.28 n.3, p.300-311, March 1985 [doi>10.1145/3166.3170]
• 6: J. Grudin , S. E. Poltrock, User interface design in large corporations: coordination and communication across disciplines, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.197-203, March 1989
• 7: Hackman, J.R., and Morris, C.G. Group tasks, group interaction process, and group performance effectiveness: A review and proposed integration. In Berkowitz, L., (Ed.), Advances in Experimental Social Psychology, V ol. 8, Academic Press, New York, 1975.
• 8: International Business Machines Corporation. Systems Ap. plication Architecture, Common User Access, Guide to User Interface Design, (SC34-4289), 1991.
• 9: International Standards Organization. Working Paper of ISO 9241 Part 10, 'Dialogue Principles, Version 2 , ISO}TC 159/SC4}WG5 N155, 1990.
• 10: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 11: Jeffries, R. J. Personal communication, Sept. 1991.
• 12: Jorgensen, A.K. Thinking-aloud in user interface design: A method promoting cognitive ergonomics. Ergonomics, 33, 4, 1990, pp. 501-507.
• 13: Karat, C. Cost-justifying human factors support on development projects. Human Factors Society Bulletin, 35, 3, 1992, pp. 1-4.
• 14: Karat, C. Cost-benefit and business case analysis of usability engineering. ACM SIGCIll Conference on Human Factors in Computing Systems, New Orleans, LA, April 28-May2, Tutorial Notes.
• 15: Karat, C. Cost-benefit analysis of usability engineering techniques. In Proceedings of the HFS Society, (Orlando, FL, Oct., 1990), pp. 839-843.
• 16: Clayton Lewis , Peter G. Polson , Cathleen Wharton , John Rieman, Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.235-242, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97279]
• 17: McGrath, J.E. Groups: interaction and Performance Prentice- Hall, Englewood Cliffs, N.J., 1984.
• 18: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 19: Jakob Nielsen, Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.), p.394-401, September 1989, Boston, Massachusetts, United States
• 20: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 21: Neal Whitten, Managing software development projects: formula for success, John Wiley & Sons, Inc., New York, NY, 1990
• 22: Cathleen Wharton , Janice Bradford , Robin Jeffries , Marita Franzke, Applying cognitive walkthroughs to more complex user interfaces: experiences, issues, and recommendations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.381-388, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142864]
• 23: Peter C. Wright , Andrew F. Monk, The use of think-aloud evaluation methods in design, ACM SIGCHI Bulletin, v.23 n.1, p.55-57, Jan. 1991 [doi>10.1145/122672.122685]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

General Terms:
Design, Human Factors, Performance

Keywords:
cost-effectiveness, empirical testing, problem severity, scenarios, walkthroughs