A tool for creating predictive performance models from user interface demonstrations

ABSTRACT

A central goal of many user interface development tools has been to make the construction of high quality interfaces easy enough that iterative design approaches could be a practical reality. In the last 15 years significant advances in this regard have been achieved. However, the evaluation portion of the iterative design process has received relatively little support from tools. Even though advances have also been made in usability evaluation methods, nearly all evaluation is still done “by hand,” making it more expensive and difficult than it might be. This paper considers a partial implementation of the CRITIQUE usability evaluation tool that is being developed to help remedy this situation by automating a number of evaluation tasks. This paper will consider techniques used by the system to produce predictive models (keystroke level models and simplified GOMS models) from demonstrations of sample tasks in a fraction of the time needed by conventional handcrafting methods. A preliminary comparison of automatically generated models with models created by an expert modeler show them to produce very similar predictions (within 2%). Further, because they are automated, these models promise to be less subject to human error and less affected by the skill of the modeler.

REFERENCES

• BEAR96: Beard, David V., Smith, Dana K. & Denelsbeck, Kevin M. (1996). "Quick and Dirty GOMS: A Case Study of Computed Tomography", Human-Computer Interaction, v.11 n.2 p.157-180.
• BUXT83: W. Buxton , M. R. Lamb , D. Sherman , K. C. Smith, Towards a comprehensive user interface management system, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.35-42, July 25-29, 1983, Detroit, Michigan, United States
• BYRN94: Michael D. Byrne , Scott D. Wood , James D. Foley , David E. Kieras , Piyawadee Noi Sukaviriya, Automating interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.232-237, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191752]
• CARD83: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• EDWA97: W. Keith Edwards , Scott E. Hudson , Joshua Marinacci , Roy Rodenstein , Thomas Rodriguez , Ian Smith, Systematic output modification in a 2D user interface toolkit, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.151-158, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263537]
• ELKE91: Jay Elkerton , Susan L. Palmiter, Designing help using a GOMS model: an information retrieval evaluation, Human Factors, v.33 n.2, p.185-204, April 1991
• GRAY93: Gray, Wayne D., John, Bonnie E. & Atwood, Michael E. (1993) ."Project Ernestine: Validating a GOMS Analysis for Predicting and Explaining Real-World Task Performance", Human-Computer Interaction, v.8 (3), pp.237-309.
• GONG94: Richard Gong , David Kieras, A validation of the GOMS model methodology in the development of a specialized, commercial software application, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.351-357, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191782]
• HENR90: Tyson R. Henry , Scott E. Hudson , Gary L. Newell, Integrating gesture and snapping into a user interface toolkit, Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, p.112-122, October 03-05, 1990, Snowbird, Utah, United States [doi>10.1145/97924.97938]
• HUDS95a: Scott E. Hudson , John T. Stasko, Animation support in a user interface toolkit: flexible, robust, and reusable abstractions, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.57-67, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168648]
• HUDS95b: Scott E. Hudson , Ian Smith, Ultra-lightweight constraints, Proceedings of the 9th annual ACM symposium on User interface software and technology, p.147-155, November 06-08, 1996, Seattle, Washington, United States [doi>10.1145/237091.237112]
• MYER90: Brad A. Myers, A new model for handling input, ACM Transactions on Information Systems (TOIS), v.8 n.3, p.289-320, July 1990 [doi>10.1145/98188.98204]
• MYER96: Brad A. Myers , David S. Kosbie, Reusable hierarchical command objects, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.260-267, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238526]
• JOHN94: Bonnie E. John, Toward a deeper comparison of methods: a reaction to Nielsen & Phillips and new data, Conference companion on Human factors in computing systems, p.285-286, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260495]
• JOHN95: Bonnie John, Why GOMS?, interactions, v.2 n.4, p.80-89, Oct. 1995 [doi>10.1145/225362.225374]
• JOHN96a: Bonnie E. John , David E. Kieras, The GOMS family of user interface analysis techniques: comparison and contrast, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.4, p.320-351, Dec. 1996 [doi>10.1145/235833.236054]
• JOHN96b: Bonnie E. John , David E. Kieras, Using GOMS for user interface design and evaluation: which technique?, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.4, p.287-319, Dec. 1996 [doi>10.1145/235833.236050]
• KIER93: Kieras, D. E. (1993) "Using the Keystroke-Level Model to Estimate Execution Times", Unpublished Report, University of Michigan.
• KIER96: Kieras, D. E. (1996) "Guide to GOMS model usability evaluations using NGOMSL", The Handbook of Human- Computer Interaction, M. Helander and T.Landauer (Eds.), 2nd ed. North-Holland Amsterdam.
• KIER: Kieras, D. "A Guide to GOMS Model Usability Evaluation using NGOMSL", The Handbook of Human- Computer Interaction, (Second Edition). M. Helander & T. Landauer (Eds.), Amsterdam: North-Holland (in press).
• KOSB94: Kosbie, D. S. & John, B. E. (1994). "Hierarchical Event Histories and GOMS", Poster at the Human-Computer Interaction Consortium Winter Workshop (January, 1994, Frasier, Colorado.)
• LAND95: Thomas K. Landauer, The trouble with computers, MIT Press, Cambridge, MA, 1995
• LANE93: Lane, D. M., Napier, H. A., Batsell, R. R. & Naman, J. L. (1993). "Predicting the Skilled Use of Hierarchical Menus with the Keystroke-Level Model", Human-Computer Interaction, v.8(2), pp. 185-192.
• LECE98: Andreas Lecerof , Fabio Paternò, Automatic Support for Usability Evaluation, IEEE Transactions on Software Engineering, v.24 n.10, p.863-888, October 1998 [doi>10.1109/32.729686]
• LEE89: A. Y. Lee , P. G. Polson , W. A. Bailey, Learning and transfer of measurement tasks, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.115-120, March 1989
• LERC89: F. J. Lerch , M. M. Mantei , J. R. Olson, Skilled financial planning: the cost of translating ideas into action, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.121-126, March 1989
• LEWI94: Lewis, C. & Rieman, J. (1994). Task-centered User Interface Design: A Practical Iintroduction. Shareware book available at ftp'cs'c~l~rad~'edu/pub/cs/distribs/ clewis/HCI- Design-Book.
• MOLI90: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• NIEL94: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• OLSE88: Dan R. Olsen , Bradley W. Halversen, Interface usage measurements in a user interface management system, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.102-108, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62420]
• OLSO90: Olson, J. R. & Olson, G. M. (1990). "The Growth of Cognitive Modeling in Human-Computer Interaction since GOMS", Human-Computer Interaction, v.5,221-265.
• SEAR95: Andrew Sears, AIDE: a step toward metric-based interface development tools, Proceedings of the 8th annual ACM symposium on User interface and software technology, p.101-110, November 15-17, 1995, Pittsburgh, Pennsylvania, United States [doi>10.1145/215585.215704]
• WHAR94: Cathleen Wharton , John Rieman , Clayton Lewis , Peter Polson, The cognitive walkthrough method: a practitioner's guide, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

General Terms:
Design, Performance, Theory

Keywords:
GOMS, event logs, task modeling, tool support for evaluation, toolkits