An application-independent system for visualizing user operation history

ABSTRACT

A history-of-user-operations function helps make applications easier to use. For example, users may have access to an operation history list in an application to undo or redo a past operation. To provide an overview of a long operation history and help users find target interactions or application states quickly, visual representations of operation history have been proposed. However, most previous systems are tightly integrated with target applications and difficult to apply to new applications. We propose an application-independent method that can visualize the operation history of arbitrary GUI applications by monitoring the input and output GUI events from outside of the target application. We implemented a prototype system that visualizes operation sequences of generic Java Awt/Swing applications using an annotated comic strip metaphor. We tested the system with various applications and present results from a user study.

REFERENCES

• 1: Maneesh Agrawala , Doantam Phan , Julie Heiser , John Haymaker , Jeff Klingner , Pat Hanrahan , Barbara Tversky, Designing effective step-by-step assembly instructions, ACM SIGGRAPH 2003 Papers, July 27-31, 2003, San Diego, California
• 2: Maneesh Agrawala , Chris Stolte, Rendering effective route maps: improving usability through generalization, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.241-249, August 2001 [doi>10.1145/383259.383286]
• 3: Jackie Assa , Yaron Caspi , Daniel Cohen-Or, Action synopsis: pose selection and illustration, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 4: Baudisch, P., Cutrell, E., and Robertson, G. High-Density Cursor: A Visualization Technique that Helps Users Keep Track of Fast-Moving Mouse Cursors. In Proceedings of INTERACT, pp. 236--243, 2003.
• 5: Patrick Baudisch , Desney Tan , Maxime Collomb , Dan Robbins , Ken Hinckley , Maneesh Agrawala , Shengdong Zhao , Gonzalo Ramos, Phosphor: explaining transitions in the user interface using afterglow effects, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166280]
• 6: Lawrence Bergman , Vittorio Castelli , Tessa Lau , Daniel Oblinger, DocWizards: a system for authoring follow-me documentation wizards, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095067]
• 7: Thomas Berlage, A selective undo mechanism for graphical user interfaces based on command objects, ACM Transactions on Computer-Human Interaction (TOCHI), v.1 n.3, p.269-294, Sept. 1994 [doi>10.1145/196699.196721]
• 8: Anastasia Bezerianos , Pierre Dragicevic , Ravin Balakrishnan, Mnemonic rendering: an image-based approach for exposing hidden changes in dynamic displays, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166279]
• 9: Freeman, W.T., and Zhang, H. Shape-Time Photography. In Proceedings of CVPR, pp. 151--157, 2003.
• 10: Dan B Goldman , Brian Curless , David Salesin , Steven M. Seitz, Schematic storyboarding for video visualization and editing, ACM SIGGRAPH 2006 Papers, July 30-August 03, 2006, Boston, Massachusetts
• 11: François Guimbretiére , Morgan Dixon , Ken Hinckley, ExperiScope: an analysis tool for interaction data, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240826]
• 12: Takeo Igarashi , Satoshi Matsuoka , Hidehiko Tanaka, Teddy: a sketching interface for 3D freeform design, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.409-416, July 1999 [doi>10.1145/311535.311602]
• 13: Irani, M., and Anandan, P. Video Indexing Based on Mosaic Representations. IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 86, No. 5, pp. 905--921, 1998.
• 14: Victor Kaptelinin , Timo Mäntylä , Jan Åström, Transient visual cues for scrolling: an empirical study, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506513]
• 15: Kawagishi, Y., Hatsuyama, K., and Kondo, K. Cartoon Blur: Non-Photorealistic Motion Blur. In Proceedings of CGI, pp. 276--281, 2003.
• 16: B. Kim , I. Essa, Video-based nonphotorealistic and expressive illustration of motion, Proceedings of the Computer Graphics International 2005, p.32-35, June 02-04, 2005
• 17: David Kurlander , Steven Feiner, A history-based macro by example system, Proceedings of the 5th annual ACM symposium on User interface software and technology, p.99-106, November 15-18, 1992, Monteray, California, United States [doi>10.1145/142621.142633]
• 18: Henry Lieberman, Mondrian: a teachable graphical editor, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 19: Greg Little , Tessa A. Lau , Allen Cypher , James Lin , Eben M. Haber , Eser Kandogan, Koala: capture, share, automate, personalize business processes on the web, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240767]
• 20: M. Massey , W. Bender, Salient stills: process and practice, IBM Systems Journal, v.35 n.3-4, p.557-573, 1996
• 21: Maic Masuch , Stefan Schlechtweg , Ronny Schulz, Speedlines: depicting motion in motionless pictures, ACM SIGGRAPH 99 Conference abstracts and applications, p.277, August 08-13, 1999, Los Angeles, California, United States [doi>10.1145/311625.312369]
• 22: C. Meng , M. Yasue , A. Imamiya , X. Mao, Visualizing Histories for Selective Undo and Redo, Proceedings of the Third Asian Pacific Computer and Human Interaction, p.459, July 15-17, 1998
• 23: Francesmary Modugno , Brad A. Myers, Pursuit: graphically representing programs in a demonstrational visual shell, Conference companion on Human factors in computing systems, p.455-456, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260464]
• 24: Brad A. Myers , Richard G. McDaniel , Robert C. Miller , Alan S. Ferrency , Andrew Faulring , Bruce D. Kyle , Andrew Mickish , Alex Klimovitski , Patrick Doane, The Amulet Environment: New Models for Effective User Interface Software Development, IEEE Transactions on Software Engineering, v.23 n.6, p.347-365, June 1997 [doi>10.1109/32.601073]
• 25: Elizabeth D. Mynatt , Takeo Igarashi , W. Keith Edwards , Anthony LaMarca, Flatland: new dimensions in office whiteboards, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.346-353, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303108]
• 26: Jun Rekimoto, Time-machine computing: a time-centric approach for the information environment, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.45-54, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322582]
• 27: Su, S. Visualizing, Editing, and Inferring Structure in 2D Graphics, UIST 2007 Doctoral Symposium, 2007.
• 28: Yukinobu Taniguchi , Akihito Akutsu , Yoshinobu Tonomura, PanoramaExcerpts: extracting and packing panoramas for video browsing, Proceedings of the fifth ACM international conference on Multimedia, p.427-436, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266396]
• 29: Laura Teodosio , Walter Bender, Salient video stills: content and context preserved, Proceedings of the first ACM international conference on Multimedia, p.39-46, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166270]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Design, Human Factors

Keywords:
diagrams, operation history, program visualization, storyboards, summarization