Metrics for analyzing rich session histories

ABSTRACT

To be most useful, evaluation metrics should be based on detailed observation and effective analysis of a full spectrum of system use. Because observation is costly, ideally we want a system to provide in-depth data collection with allied analyses of the key user interface elements. We have developed a visualization and analysis platform [1] that automatically records user actions and states at a high semantic level [2 and 3], and can be directly restored to any state. Audio and text annotations are collected and indexed to states, allowing users to comment on their current situation as they work, and/or as they review the session. These capabilities can be applied to usability evaluation of the system, describing problems they encountered, or to suggest improvements to the environment. Additionally, computed metrics are provided at each state [3, 4, and 5]. We believe that the metrics and the associated history data will allow us to deduce patterns of data exploration, to compare users, to evaluate tools, and to understand in a more automated approach the usability of the visualization system as a whole.

REFERENCES

• 1: Gee, A. G., Li, H., Yu, M., Smrtic, M. B., Cvek, U., Goodell, H., Gupta, V., Lawrence, C., Zhou, J., Chiang, C.-H. and Grinstein, G. G., Universal Visualization Platform, in Visualization and Data Analysis 2005, (San Jose, California, USA, 2005), SPIE, 274--283.
• 2: Goodell, H., Chiang, C.-H. and Grinstein, G. G. Taxonomized Automated Monitoring of Interactive Visualization, Computer Science Department, University of Massachusetts at Lowell, Lowell, Massachusetts, 2006.
• 3: Howard Goodell , Chih-Hung Chiang , Curran Kelleher , Alex Baumann , Georges Grinstein, Collecting and Harnessing Rich Session Histories, Proceedings of the conference on Information Visualization, p.117-123, July 05-07, 2006 [doi>10.1109/IV.2006.33]
• 4: J. P. Lee , Georges Grinstein, An Architecture for Retaining and Analyzing Visual Explorations of Databases, Proceedings of the 6th conference on Visualization '95, p.101, October 29-November 03, 1995
• 5: Lee, J. P. A systems and process model for data exploration Computer Science, Dissertation, University of Massachusetts at Lowell, Lowell, Massachusetts, USA, 1998, 245.
• 6: Rooden, M. J. T., Ed. (1998). Thinking About Thinking Aloud. Contemporary Ergonomics. London, Taylor and Francis.
• 7: Boren, M. T. and J. Ramey (2000). "Thinking Aloud: Reconciling Theory and Practice." IEEE Transactions on Professional Communication 43(3): 261--278
• 8: Bowers, V. A. (1990). Concurrent Versus Retrospective Verbal Protocols for Comparing Window Usability. Industrial Engineering and Operations Research. Blacksburg, Virginia, USA, Virginia Polytechnic Institute and State University. Doctorate thesis.
• 9: Jun Rekimoto, Time-machine computing: a time-centric approach for the information environment, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.45-54, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322582]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Graphical user interfaces (GUI); User interface management systems (UIMS); Theory and methods; Standardization

General Terms:
Documentation, Experimentation, Human Factors, Measurement, Standardization, Verification

Keywords:
session history analysis, session history visualization, user monitoring