Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems

ABSTRACT

Automatic recording of user behavior within a system (instrumentation) to develop and test theories has a rich history in psychology and system design. Often, researchers analyze instrumented behavior in isolation from other data. The problem with collecting instrumented behaviors without attitudinal, demographic, and contextual data is that researchers have no way to answer the 'why' behind the 'what'. We have combined the collection and analysis of behavioral instrumentation with other HCI methods to develop a system for Tracking Real-Time User Experience (TRUE). Using two case studies as examples, we demonstrate how we have evolved instrumentation methodology and analysis to extensively improve the design of video games. It is our hope that TRUE is adopted and adapted by the broader HCI community, becoming a useful tool for gaining deep insights into user behavior and improvement of design for other complex systems.

REFERENCES

• 1: Badre, A.N. and Santos, P. J. CHIME: A Knowledge-Based Computer-Human Interaction Monitoring Engine. Tech Rept. GIT-GVU-91-06, 1991.
• 2: Balbo, S. EMA: Automatic Analysis Mechanism for the Ergonomic Evaluation of User Interfaces. Tech Rept. CSIRO-DIT 96/44, 1996.
• 3: Elizabeth Chang , Tharam S. Dillon, Automated Usability Testing, Proceedings of the IFIP TC13 Interantional Conference on Human-Computer Interaction, p.77-84, July 14-18, 1997
• 4: Davis, J., Steury, K., and Pagulayan, R. A survey method for assessing perceptions of a game: The consumer playtest in game design. Game Studies: The International Journal of Computer Game Research, 5 (2005). http://www.gamestudies.org/0501/davis_steury_pagulayan/
• 5: Joseph S. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Intellect Books, Exeter, UK, 1999
• 6: David M. Hilbert , David F. Redmiles, Extracting usability information from user interface events, ACM Computing Surveys (CSUR), v.32 n.4, p.384-421, Dec. 2000 [doi>10.1145/371578.371593]
• 7: Lorin Hochstein , Victor R. Basili , Marvin V. Zelkowitz , Jeffrey K. Hollingsworth , Jeff Carver, Combining self-reported and automatic data to improve programming effort measurement, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 8: Hugh Beyer , Karen Holtzblatt, Contextual design: defining customer-centered systems, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 9: Amy Hurst , Scott E. Hudson , Jennifer Mankoff, Dynamic detection of novice vs. skilled use without a task model, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240669]
• 10: Melody Y. Ivory , Marti A Hearst, The state of the art in automating usability evaluation of user interfaces, ACM Computing Surveys (CSUR), v.33 n.4, p.470-516, December 2001 [doi>10.1145/503112.503114]
• 11: Kort, J., Steen, M.G.D., de Poot, H., ter Hofte, H., and Mulder, I. Studying usage of complex applications. In Proc. Measuring Behav. 2005, Noldus Information Technology (2005), 266--269.
• 12: Jonas Löwgren , Tommy Nordqvist, Knowledge-based evaluation as design support for graphical user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.181-188, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142786]
• 13: Medlock, M.C., Wixon, D., Terrano, M., Romero, R.L., and Fulton, B. Using the RITE method to improve products: a definition and a case study. In Proc. UPA 2002, UPA (2002).
• 14: Miles, M.B. and Huberman, M.A. Qualitative Data Analysis: An Expanded Source Book. Sage Publications, Thousand Oaks, CA, 1994.
• 15: Misanchuk, E.R. and Schwier, R. Representing interactive multimedia and hypermedia audit trails. Journal of Educational Multimedia and Hypermedia 1, 3 (1992), 355--372.
• 16: Alan S. Neal , Roger M. Simons, Playback: A method for evaluating the usability of software and its documentation, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.78-82, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801585]
• 17: Pagulayan, R., Gunn, D., and Romero, R. A Gameplay-Centered Design Framework for Human Factors in Games. In W. Karwowski (Ed.), 2nd Edition of International Encyclopedia of Ergonomics and Human Factors, Taylor & Francis (2006), 1314--1319.
• 18: Pagulayan, R., Keeker, K., Fuller, T., Wixon, D., and Romero, R. User-centered design in games (revision). In J. Jacko and A. Sears (Eds.), Handbook for Human-Computer Interaction in Interactive Systems, Lawrence Erlbaum Associates (In press).
• 19: Rea, L.M. and Parker. R.A. Designing and Conducting Survey Research: A Comprehensive Guide (3rd ed.). Jossey-Bass Publishers, CA, USA, 2005.
• 20: Reeves, T.C. and Hedberg, J.G. Interactive Learning Systems Evaluation. Educational Technology Publications, Englewood Cliffs, NJ, 2003.
• 21: Karen Renaud , Phil Gray, Making sense of low-level usage data to understand user activities, Proceedings of the 2004 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, p.115-124, October 04-06, 2004, Stellenbosch, Western Cape, South Africa
• 22: Skinner, B.F. On the rate of formation of a conditioned reflex. Journal of General Psychology, 7 (1932), 274--86.
• 23: Skinner, B.F. The behavior of organisms: An experimental analysis. Appleton-Century, NY, USA, 1938.
• 24: Thompson, C. Halo 3: How Microsoft Labs Invented a New Science of Play. Wired, 15.09 (2007). http://www.wired.com/gaming/virtualworlds/magazine/15-09/ff_halo
• 25: Thorndike, E.L. Animal intelligence: An experimental study of the associative processes in animals. Psychological Review Monograph Supplement 2, 4 (1898), Whole No. 8.
• 26: Thorndike, E.L. Animal Intelligence. Macmillan, NY, USA, 1911.
• 27: John Whiteside , Norman Archer , Dennis Wixon , Michael Good, How do people really use text editors?, Proceedings of the SIGOA conference on Office information systems, p.29-40, June 21-23, 1982, Philadelphia, Pennsylvannia, United States

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

General Terms:
Human Factors

Keywords:
instrumentation, true, usability, user initiated events