Extracting usability information from user interface events

ABSTRACT

Modern window-based user interface systems generate user interface events as natural products of their normal operation. Because such events can be automatically captured and because they indicate user behavior with respect to an application's user interface, they have long been regarded as a potentially fruitful source of information regarding application usage and usability. However, because user interface events are typically voluminos and rich in detail, automated support is generally required to extract information at a level of abstraction that is useful to investigators interested in analyzing application usage or evaluating usability. This survey examines computer-aided techniques used by HCI practitioners and researchers to extract usability-related information from user interface events. A framework is presented to help HCI practitioners and researchers categorize and compare the approaches that have been, or might fruitfully be, applied to this problem. Because many of the techniques in the research literature have not been evaluated in practice, this survey provides a conceptual evaluation to help identify some of the relative merits and drawbacks of the various classes of approaches. Ideas for future research in this area are also presented. This survey addresses the following questions: How might user interface events be used in evaluating usability? How are user interface events related to other forms of usability data? What are the key challenges faced by investigators wishing to exploit this data? What approaches have been brought to bear on this problem and how do they compare to one another? What are some of the important open research questions in this area?

REFERENCES

• 1: ABBOTT, A. 1990. A primer on sequence methods. Organ. Sci. 4.
• 2: AGRAWAL, R., ARNING, A., BOLLINGER, T., MEHTA, M., SHAFER,J.,AND SRIKANT, R. 1996. The Quest data mining system. In Proceedings of the 2nd International Conference on Knowledge Discovery in Databases and Data Mining.
• 3: Alfred V. Aho , Brian W. Kernighan , Peter J. Weinberger, The AWK programming language, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1987
• 4: ALLISON,P.D.AND LIKER, J. K., 1982. Analyzing sequential categorical data on dyadic interaction: A comment on Gottman. Psychological Bulletin 2.
• 5: AQUEDUCT SOFTWARE. 1998. AppScope Web Pages. URL: http://www.aqueduct.com/.
• 6: BADRE,A.N.AND SANTOS, P. J. 1991a. CHIME: A Knowledge-Based Computer-Human Interaction Monitoring Engine. Tech Rept. GIT-GVU-91- 06.
• 7: BADRE,A.N.AND SANTOS, P. J. 1991b. A Knowledge-Based System for Capturing Human-Computer Interaction Events: CHIME. Tech. Rept. GIT-GVU- 91-21.
• 8: BADRE,A.N.,GUZDIAL, M., HUDSON,S.E.,AND SANTOS, P. J. 1995. A user interface evaluation environment using synchronized video, visualizations, and event trace data. J. of Software Qual. 4.
• 9: BAECKER,R.M,GRUDIN, J., BUXTON,W.A.S.,AND GREENBERG, S., Eds. 1995. Readings in Human-Computer Interaction: Toward the Year 2000. Morgan Kaufmann, San Mateo, CA.
• 10: BALBO, S. 1996. EMA: Automatic Analysis Mechanism for the Ergonomic Evaluation of User Interfaces. CSIRO Tech. rep.
• 11: Peter C. Bates, Debugging heterogeneous distributed systems using event-based models of behavior, ACM Transactions on Computer Systems (TOCS), v.13 n.1, p.1-31, Feb. 1995 [doi>10.1145/200912.200913]
• 12: Victoria Bellotti, A framework for assessing applicability of HCI techniques, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.213-218, August 27-31, 1990
• 13: W. Buxton , M. R. Lamb , D. Sherman , K. C. Smith, Towards a comprehensive user interface management system, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.35-42, July 25-29, 1983, Detroit, Michigan, United States
• 14: Elizabeth Chang , Tharam S. Dillon, Automated Usability Testing, Proceedings of the IFIP TC13 Interantional Conference on Human-Computer Interaction, p.77-84, July 14-18, 1997
• 15: Jolly Chen, Providing intrinsic support for user interface monitoring, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.415-420, August 27-31, 1990
• 16: COOK,J.E.AND WOLF, A. L. 1994. Toward metrics for process validation. In Proceedings of ICSP '94.
• 17: Jonathan E. Cook , Alexander L. Wolf, Automating process discovery through event-data analysis, Proceedings of the 17th international conference on Software engineering, p.73-82, April 24-28, 1995, Seattle, Washington, United States [doi>10.1145/225014.225021]
• 18: COOK,J.E.AND WOLF, A. L. 1997. Software Process Validation: Quantitatively Measuring the Correspondence of a Process to a Model. Tech. Rep. CU-CS-840-97, Dept. of Computer Science, Univ. of Colorado at Boulder.
• 19: COOK, R., KAY, J., RYAN,G.,AND THOMAS, R. C. 1995. A toolkit for appraising the long-term usability of a text editor. Software Qual. J. 4,2.
• 20: CUOMO, D. L. 1994. Understanding the applicability of sequential data analysis techniques for analysing usability data. In Usability Laboratories Special Issue of Behavior and Information Technology, J. Nielsen, Ed., vol. 13, no.1 & 2.
• 21: Allen Cypher, EAGER: programming repetitive tasks by example, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.33-39, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108850]
• 22: Allen Cypher , Daniel C. Halbert , David Kurlander , Henry Lieberman , David Maulsby , Brad A. Myers , Alan Turransky, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 23: Mark Dodge , Craig Stinson, Running Microsoft Excel 2000, Microsoft Press, Redmond, WA, 1999
• 24: Ann Doubleday , Michele Ryan , Mark Springett , Alistair Sutcliffe, A comparison of usability techniques for evaluating design, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.101-110, August 18-20, 1997, Amsterdam, The Netherlands [doi>10.1145/263552.263583]
• 25: Bruce Elgin, Subjective usability feedback from the field over a network, Conference companion on Human factors in computing systems, p.340, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223709]
• 26: ERGOLIGHT USABILITY SOFTWARE. 1998. Operation Recording Suite (EORS) and Usability Validation Suite (EUVS) Web pages. URL: http:/ /www.ergolight.co.il/.
• 27: FARAONE,S.V.AND DORFMAN, D. D. 1987. Lag sequential analysis: Robust statistical methods. Psychological Bulletin 101.
• 28: Don Cohen , Martin S. Feather , K. Narayanaswamy , Stephen S. Fickas, Automatic monitoring of software requirements, Proceedings of the 19th international conference on Software engineering, p.602-603, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253493]
• 29: S. Fickas , M. S. Feather, Requirements monitoring in dynamic environments, Proceedings of the Second IEEE International Symposium on Requirements Engineering, p.140, March 27-29, 1995
• 30: Janet Finlay , Michael D. Harrison, Pattern recognition and interaction models, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.149-154, August 27-31, 1990
• 31: Carolanne Fisher, Advancing the study of programming with computer-aided protocol analysis, Empirical studies of programmers: second workshop, Ablex Publishing Corp., Norwood, NJ, 1987
• 32: FISHER, C. 1991. Protocol Analyst's Workbench: Design and Evaluation of Computer-Aided Protocol Analysis. Unpublished Ph.D. thesis, Carnegie Mellon University, Dept. of Psychology, Pittsburgh, PA.
• 33: Carolanne Fisher , Penelope Sanderson, Exploratory sequential data analysis: exploring continuous observational data, interactions, v.3 n.2, p.25-34, March 1996 [doi>10.1145/227181.227185]
• 34: FITTS, P. M. 1964. Perceptual motor skill learning. In Categories of human learning, A. W. Melton, Ed. Academic Press, New York, NY.
• 35: FULL CIRCLE SOFTWARE. 1998. Talkback Web pages. URL: http://www.fullsoft.com/.
• 36: GIRGENSOHN, A., REDMILES,D.F.,AND SHIPMAN, F. M. III. 1994. Agent-based support for communication between developers and users in software design. In Proceedings of the Knowledge-Based Software Engineering Conference '94. Monterey, CA, USA.
• 37: Danny Goodman, The complete HyperCard 2.2 handbook (4th ed.), Random House Inc., New York, NY, 1993
• 38: GOTTMAN,J.M.AND ROY, A. K. 1990. Sequential analysis: A guide for behavioral researchers. Cambridge University Press, Cambridge, England.
• 39: GRUDIN, J. 1992. Utility and usability: Research issues and development contexts. Interacting with comput. 4,2.
• 40: GUZDIAL, M. 1993. Deriving Software Usage Patterns from Log Files. Tech. Rept. GIT-GVU-93- 41.
• 41: GUZDIAL,M,WALTON, C., KONEMANN, M., AND SOLOWAY, E. 1993. Characterizing Process Change Using Log File Data. Tech. Rep. GIT-GVU-93- 44.
• 42: GUZDIAL, M., SANTOS, P., BADRE, A., HUDSON,S.,AND GRAY, M. 1994. Analyzing and visualizing log files: Acomputational science of usability. In Presented at HCI Consortium Workshop.
• 43: H. Rex Hartson , José C. Castillo , John Kelso , Wayne C. Neale, Remote evaluation: the network as an extension of the usability laboratory, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.228-235, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238511]
• 44: Martin G. Helander , Thomas K. Landauer , P. Prabhu , Prasad V. Prabhu, Handbook of Human-Computer Interaction, Elsevier Science Inc., New York, NY, 1997
• 45: HEWLETT PACKARD. 1998. Application Response Measurement API. URL: http://www.hp.com/ openview/rpm/arm/.
• 46: David M. Hilbert , David F. Redmiles, An approach to large-scale collection of application usage data over the Internet, Proceedings of the 20th international conference on Software engineering, p.136-145, April 19-25, 1998, Kyoto, Japan
• 47: David M. Hilbert , David F. Redmiles, Agents for collecting application usage data over the Internet, Proceedings of the second international conference on Autonomous agents, p.149-156, May 10-13, 1998, Minneapolis, Minnesota, United States [doi>10.1145/280765.280793]
• 48: HILBERT, D. M., ROBBINS,J.E.,AND REDMILES,D.F., 1997. Supporting Ongoing User Involvement in Development via Expectation-Driven Event Monitoring. Tech Report UCI-ICS-97-19, Dept. of Information and Computer Science, Univ. of California, Irvine.
• 49: D. S. Hirschberg, A linear space algorithm for computing maximal common subsequences, Communications of the ACM, v.18 n.6, p.341-343, June 1975 [doi>10.1145/360825.360861]
• 50: HOIEM,D.E.AND SULLIVAN, K. D. 1994. Designing and using integrated data collection and analysis tools: challenges and considerations. In Usability Laboratories Special Issue of Behavior and Information Technology, J. Nielsen, Ed., vol. 13,no.1 &2.
• 51: H. U. Hoppe, Task-oriented parsing - a diagnostic method to be used adaptive systems, Proceedings of the SIGCHI conference on Human factors in computing systems, p.241-247, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57208]
• 52: Bonnie E. John , David E. Kieras, The GOMS family of user interface analysis techniques: comparison and contrast, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.4, p.320-351, Dec. 1996 [doi>10.1145/235833.236054]
• 53: Bonnie E. John , David E. Kieras, Using GOMS for user interface design and evaluation: which technique?, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.4, p.287-319, Dec. 1996 [doi>10.1145/235833.236050]
• 54: Judy Kay , Richard C. Thomas, Studying long-term system use, Communications of the ACM, v.38 n.7, p.61-69, July 1995 [doi>10.1145/213859.214799]
• 55: David S. Kosbie , Brad A. Myers, Extending Programming by Demonstration with Hierarchical Event Histories, Selected papers from the 4th International Conference on Human-Computer Interaction, p.128-139, August 02-05, 1994
• 56: Balachander Krishnamurthy , David S. Rosenblum, Yeast: A General Purpose Event-Action System, IEEE Transactions on Software Engineering, v.21 n.10, p.845-857, October 1995 [doi>10.1109/32.469456]
• 57: Andreas Lecerof , Fabio Paternò, Automatic Support for Usability Evaluation, IEEE Transactions on Software Engineering, v.24 n.10, p.863-888, October 1998 [doi>10.1109/32.729686]
• 58: LEE, B. 1996. Remote diagnostics and product lifecycle monitoring for high-end appliances: a new Internet-based approach utilizing intelligent software agents. In Proceedings of the Appliance Manufacturer Conference.
• 59: Rita F. Lewis , Bill Fishman , Mark Stone, MAC OS in a Nutshell, O'Reilly & Associates, Inc., Sebastopol, CA, 2000
• 60: Jonas Löwgren , Tommy Nordqvist, Knowledge-based evaluation as design support for graphical user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.181-188, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142786]
• 61: MACLEOD,M.AND RENGGER, R. 1993. The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation. In Proceedings of HCI '93.
• 62: MANSOURI-SAMANI,M.AND SLOMAN, M. 1997. GEM: A generalized event monitoring language for distributed systems. IEE/BCS/IOP Distributed Syst. Eng. J. 4,2.
• 63: MERCURY INTERACTIVE. 1998. WinRunner and XRunner Web Pages. URL: http://www.merc-int.com/.
• 64: MORAN, T. P. 1981. The command language grammar: A representation for the user interface of interactive computer systems. Int. J. of Man- Machine Studies, 15.
• 65: Alan S. Neal , Roger M. Simons, Playback: A method for evaluating the usability of software and its documentation, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.78-82, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801585]
• 66: J Nielsen, A virtual protocol model for computer-human interaction, International Journal of Man-Machine Studies, v.24 n.3, p.301-312, March 1986 [doi>10.1016/S0020-7373(86)80028-1]
• 67: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 68: Jakob Nielsen , Robert L. Mack, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 69: Adrian Nye , Tim O'Reilly, X toolkit intrinsics programming manual (3rd ed.), O'Reilly & Associates, Inc., Sebastopol, CA, 1993
• 70: Dan R. Olsen , Bradley W. Halversen, Interface usage measurements in a user interface management system, Proceedings of the 1st annual ACM SIGGRAPH symposium on User Interface Software, p.102-108, October 17-19, 1988, Alberta, Canada [doi>10.1145/62402.62420]
• 71: OLSON, G. M., HERBSLEB,J.D.,AND RUETER,H.H. 1994. Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques. Hum.-Comput. Interaction Special Issue on ESDA, Vol. 9.
• 72: PAYNE,S.G.AND GREEN, T. R. G. 1986. Taskaction grammars: A model of the mental representation of task languages. Hum.-Comput. Interaction, Vol. 2.
• 73: PENTLAND, B. T. 1994. A grammatical model of organizational routines. Administrative Sci. Quarterly.
• 74: PENTLAND, B. T. 1994. Grammatical models of organizational processes. Organ. Sci.
• 75: C. Petzold, Programming Windows, Microsoft Press, Redmond, WA, 1988
• 76: Peter G. Polson , Clayton Lewis , John Rieman , Cathleen Wharton, Cognitive walkthroughs: a method for theory-based evaluation of user interfaces, International Journal of Man-Machine Studies, v.36 n.5, p.741-773, May 1992 [doi>10.1016/0020-7373(92)90039-N]
• 77: Jenny Preece , Yvonne Rogers , Helen Sharp , David Benyon , Simon Holland , Tom Carey, Human-Computer Interaction, Addison-Wesley Longman Ltd., Essex, UK, 1994
• 78: David S. Rosenblum, Specifying Concurrent Systems with TSL, IEEE Software, v.8 n.3, p.52-61, May 1991 [doi>10.1109/52.88944]
• 79: Charles Rubin, Running Microsoft works, Microsoft Press, Redmond, WA, 1990
• 80: SACKETT, G. P. 1978. Observing Behavior, Vol. University Park Press, Baltimore, MD.
• 81: SANDERSON,P.M.AND FISHER, C. 1994. Exploratory sequential data analysis: foundations. Hum.- Comput. Interaction Special Issue on ESDA, 9.
• 82: Penelope Sanderson , Jay Scott , Tom Johnston , John Mainzer , Larry Watanabe , Jeff James, MacSHAPA and the enterprise of exploratory sequential data analysis (ESDA), International Journal of Human-Computer Studies, v.41 n.5, p.633-681, Nov. 1994 [doi>10.1006/ijhc.1994.1077]
• 83: Paulo J. Santos , Albert N. Badre, Automatic chunk detection in human-computer interaction, Proceedings of the workshop on Advanced visual interfaces, p.69-77, June 01-04, 1994, Bari, Italy [doi>10.1145/192309.192327]
• 84: Franz Schiele , H. Ulrich Hoppe, Inferring task structures from interaction protocols, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.567-572, August 27-31, 1990
• 85: Richard W. Selby , Adam A. Porter , Doug C. Schmidt , Jim Berney, Metric-driven analysis and feedback systems for enabling empirically guided software development, Proceedings of the 13th international conference on Software engineering, p.288-298, May 13-17, 1991, Austin, Texas, United States
• 86: Antonio C. Siochi , Roger W. Ehrich, Computer analysis of user interfaces based on repetition in transcripts of user sessions, ACM Transactions on Information Systems (TOIS), v.9 n.4, p.309-335, Oct. 1991 [doi>10.1145/119311.119312]
• 87: Antonio C. Siochi , Deborah Hix, A study of computer-supported user interface evaluation using maximal repeating pattern analysis, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.301-305, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108926]
• 88: SMILOWITZ,E.D.,DARNELL,M.J.,AND BENSON,A.E. 1994. Are we overlooking some usability testing methods? A comparison of lab, beta, and forum tests. In Usability Laboratories Special Issue of Behavior and Information Technology,J. Nielsen, Ed., Vol. 13, No.1 &2.
• 89: SUN MICROSYSTEMS. 1998. SunTest JavaStar Web Pages. URL: http://www.sun.com/suntest/.
• 90: M. Sweeney , M. Maguire , B. Shackel, Evaluating user-computer interaction: a framework, International Journal of Man-Machine Studies, v.38 n.4, p.689-711, April, 1993 [doi>10.1006/imms.1993.1032]
• 91: M. M. Taylor, Layered protocols for computer-human dialogue. 1: principles, International Journal of Man-Machine Studies, v.28 n.2-3, p.175-218, Feb/Mar 1988 [doi>10.1016/S0020-7373(88)80036-1]
• 92: M. M. Taylor, Layered protocols for computer-human dialogue. 11: some practical issues, International Journal of Man-Machine Studies, v.28 n.2-3, p.219-257, Feb/Mar 1988 [doi>10.1016/S0020-7373(88)80037-3]
• 93: Richard N. Taylor , Joelle Coutaz, Workshop on software engineering and computer-human interaction: joint research issues, Proceedings of the 16th international conference on Software engineering, p.356-357, May 16-21, 1994, Sorrento, Italy
• 94: TIBCO, 1998. HAWK Enterprise Monitor Web Pages. URL: http://www.tibco.com/.
• 95: Dana L. Uehling , Karl Wolf, User action graphing effort (UsAGE), Conference companion on Human factors in computing systems, p.290-291, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223679]
• 96: USER MODELING INC. (UM Inc.), 1998. Home Page. URL: http://um.org/.
• 97: Paul Weiler, Software for the usability lab: a sampling of current tools, Proceedings of the SIGCHI conference on Human factors in computing systems, p.57-60, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169076]
• 98: WHITEFIELD, A., WILSON,F.,AND DOWELL, J. 1991. A framework for human factors evaluation. Behav. and Inf. Technol., 10,1.
• 99: WOLF,A.L.AND ROSENBLUM, D. S. 1993. A study in software process data capture and analysis. In Proceedings of the Second International Conference on Software Process.
• 100: John Zukowski, Java AWT reference, O'Reilly & Associates, Inc., Sebastopol, CA, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
human-computer interaction, sequential data analysis, usability testing, user interface event monitoring