Iterative design and evaluation of an event architecture for pen-and-paper interfaces

ABSTRACT

This paper explores architectural support for interfaces combining pen, paper, and PC. We show how the event-based approach common to GUIs can apply to augmented paper, and describe additions to address paper's distinguishing characteristics. To understand the developer experience of this architecture, we deployed the toolkit to 17 student teams for six weeks. Analysis of the developers' code provided insight into the appropriateness of events for paper UIs. The usage patterns we distilled informed a second iteration of the toolkit, which introduces techniques for integrating interactive and batched input handling, coordinating interactions across devices, and debugging paper applications. The study also revealed that programmers created gesture handlers by composing simple ink measurements. This desire for informal interactions inspired us to include abstractions for recognition. This work has implications beyond paper - designers of graphical tools can examine API usage to inform iterative toolkit development.

REFERENCES

• 1: Marc Abrams , Constantinos Phanouriou , Alan L. Batongbacal , Stephen M. Williams , Jonathan E. Shuster, UIML: an appliance-independent XML user interface language, Proceedings of the eighth international conference on World Wide Web, p.1695-1708, May 1999, Toronto, Canada
• 2: Adapx, Mapx and Penx, 2007. http://www.adapx.com
• 3: Anoto AB, Anoto Technology, 2007. http://www.anoto.com
• 4: Rafael Ballagas , Meredith Ringel , Maureen Stone , Jan Borchers, iStuff: a physical user interface toolkit for ubiquitous computing environments, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642705]
• 5: Bastéa-Forte, M., R. B. Yeh, and S. R. Klemmer. Pointer: Multiple Collocated Display Inputs Suggests New Models for Program Design and Debugging. UIST Extended Abstracts (Posters), 2007.
• 6: Bernstein, M., A. Robinson-Mosher, R. B. Yeh, and S. R. Klemmer. Diamond's Edge: From Notebook to Table and Back Again. Ubicomp Extended Abstracts (Posters), 2006.
• 7: Philip R. Cohen , David R. McGee, Tangible multimodal interfaces for safety-critical applications, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962103]
• 8: Paul Dietz , Darren Leigh, DiamondTouch: a multi-user touch technology, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502389]
• 9: EPOS, EPOS Digital Pen, 2007. http://www.epos-ps.com
• 10: Foley, J. D. and V. L. Wallace. The Art of Natural Graphic Man-Machine Conversation. IEEE 62(4). pp. 462--71, 1974.
• 11: François Guimbretière, Paper augmented digital documents, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.51-60, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964702]
• 12: Jeremy M. Heiner , Scott E. Hudson , Kenichiro Tanaka, Linking and messaging from real paper in the Paper PDA, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.179-186, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322600]
• 13: Hourcade, J. P. and B. B. Bederson, Architecture and Implementation of a Java Package for Multiple Input Devices (MID). Technical Report, University of Maryland 1999. http://www.cs.umd.edu/hcil/mid
• 14: Scott E. Hudson , Jennifer Mankoff , Ian Smith, Extensible input handling in the subArctic toolkit, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055025]
• 15: IBM Pen Technologies, CrossPad and TransNote, 2007. http://researchweb.watson.ibm.com/electricInk
• 16: Jiang, H., R. B. Yeh, T. Winograd, and Y. Shi. DigiPost: Writing on Post-its with Digital Pens to Support Collaborative Editing Tasks on Tabletop Displays. UIST Posters, 2007.
• 17: Walter Johnson , Herbert Jellinek , Leigh Klotz, Jr. , Ramana Rao , Stuart K. Card, Bridging the paper and electronic worlds: the paper user interface, Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems, p.507-512, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169445]
• 18: Miryung Kim , Lawrence Bergman , Tessa Lau , David Notkin, An Ethnographic Study of Copy and Paste Programming Practices in OOPL, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.83-92, August 19-20, 2004 [doi>10.1109/ISESE.2004.10]
• 19: Andrew J. Ko , Brad A. Myers, Designing the whyline: a debugging interface for asking questions about program behavior, Proceedings of the SIGCHI conference on Human factors in computing systems, p.151-158, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985712]
• 20: Thomas D. LaToza , Gina Venolia , Robert DeLine, Maintaining mental models: a study of developer work habits, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134355]
• 21: LeapFrog Enterprises, FLY Pentop Computer, 2007. http://www.flypentop.com
• 22: Chunyuan Liao , François Guimbretière , Ken Hinckley, PapierCraft: a command system for interactive paper, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095074]
• 23: Chunyuan Liao , François Guimbretière , Corinna E. Loeckenhoff, Pen-top feedback for paper-based interfaces, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166285]
• 24: Livescribe Inc., Livescribe, 2007. http://www.livescribe.com
• 25: Lowagie, B. and P. Soares, iText Java-PDF Library, 2007. http://www.lowagie.com/iText
• 26: Wendy E. Mackay , Guillaume Pothier , Catherine Letondal , Kaare Bøegh , Hans Erik Sørensen, The missing link: augmenting biology laboratory notebooks, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.571992]
• 27: Microsoft, Windows Forms, 2007. http://www.windowsforms.net
• 28: Mutton, P., Java EPS Graphics2D, 2007. http://www.jibble.org/epsgraphics
• 29: Brad A. Myers , John F. Pane , Andy Ko, Natural programming languages and environments, Communications of the ACM, v.47 n.9, September 2004 [doi>10.1145/1015864.1015888]
• 30: Jeffrey Nichols , Brad A. Myers , Michael Higgins , Joseph Hughes , Thomas K. Harris , Roni Rosenfeld , Mathilde Pignol, Generating remote control interfaces for complex appliances, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572008]
• 31: Moira C. Norrie , Beat Signer , Nadir Weibel, Print-n-link: weaving the paper web, Proceedings of the 2006 ACM symposium on Document engineering, October 10-13, 2006, Amsterdam, The Netherlands [doi>10.1145/1166160.1166175]
• 32: Mary Beth Rosson , John M. Carroll, The reuse of uses in Smalltalk programming, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.3, p.219-253, Sept. 1996 [doi>10.1145/234526.234530]
• 33: Bill N. Schilit , Uttam Sengupta, Device Ensembles, Computer, v.37 n.12, p.56-64, December 2004 [doi>10.1109/MC.2004.241]
• 34: Saul Schleimer , Daniel S. Wilkerson , Alex Aiken, Winnowing: local algorithms for document fingerprinting, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872770]
• 35: Signer, B., Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Unpublished PhD, ETH Zurich, Computer Science, Zurich, 2006. http://people.inf.ethz.ch/signerb/publications/signer16218.pdf
• 36: John T. Stasko , Marc H. Brown , Blaine A. Price, Software Visualization, MIT Press, Cambridge, MA, 1997
• 37: Lisa Stifelman , Barry Arons , Chris Schmandt, The audio notebook: paper and pen interaction with structured speech, Proceedings of the SIGCHI conference on Human factors in computing systems, p.182-189, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365096]
• 38: Sun Microsystems, Swing, 2007. http://java.sun.com/javase/6/docs
• 39: Tommy Thorn, Programming languages for mobile code, ACM Computing Surveys (CSUR), v.29 n.3, p.213-239, Sept. 1997 [doi>10.1145/262009.262010]
• 40: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 41: Jacob O. Wobbrock , Andrew D. Wilson , Yang Li, Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294238]
• 42: Ron Bing Yeh , Scott Robert Klemmer, Designing interactions that combine pen, paper, and computer, Stanford University, Stanford, CA, 2008
• 43: Yeh, R. B., J. Brandt, J. Boli, and S. R. Klemmer. Interactive Gigapixel Prints: Large, Paper-based Interfaces for Visual Context and Collaboration. Ubicomp Videos, 2006.
• 44: Ron Yeh , Chunyuan Liao , Scott Klemmer , François Guimbretière , Brian Lee , Boyko Kakaradov , Jeannie Stamberger , Andreas Paepcke, ButterflyNet: a mobile capture and access system for field biology research, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124859]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping; User-centered design; Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Design, Experimentation

Keywords:
augmented paper, device ensembles, evaluation, toolkits