Papiercraft: A gesture-based command system for interactive paper

ABSTRACT

Paper persists as an integral component of active reading and other knowledge-worker tasks because it provides ease of use unmatched by digital alternatives. Paper documents are light to carry, easy to annotate, rapid to navigate, flexible to manipulate, and robust to use in varied environments. Interactions with paper documents create rich webs of annotation, cross reference, and spatial organization. Unfortunately, the resulting webs are confined to the physical world of paper and, as they accumulate, become increasingly difficult to store, search, and access. XLibris [Schilit et al. 1998] and similar systems address these difficulties by simulating paper with tablet PCs. While this approach is promising, it suffers not only from limitations of current tablet computers (e.g., limited screen space) but also from loss of invaluable paper affordances.

In this article, we describe PapierCraft, a gesture-based command system that allows users to manipulate digital documents using paper printouts as proxies. Using an Anoto [Anoto 2002] digital pen, users can draw command gestures on paper to tag a paragraph, e-mail a selected area, copy selections to a notepad, or create links to related documents. Upon pen synchronization, PapierCraft executes the commands and presents the results in a digital document viewer. Users can then search the tagged information and navigate the web of annotated digital documents resulting from interactions with the paper proxies. PapierCraft also supports real time interactions across mix-media, for example, letting users copy information from paper to a Tablet PC screen. This article presents the design and implementation of the PapierCraft system and describes user feedback from initial use.

REFERENCES

• 1: Annette Adler , Anuj Gujar , Beverly L. Harrison , Kenton O'Hara , Abigail Sellen, A diary study of work-related reading: design implications for digital reading devices, Proceedings of the SIGCHI conference on Human factors in computing systems, p.241-248, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274679]
• 2: Christine Alvarado , Randall Davis, SketchREAD: a multi-domain sketch recognition engine, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029637]
• 3: Anoto. 2002. Development guide for Service Enabled by Anoto Functionality.
• 4: Toshifumi Arai , Dietmar Aust , Scott E. Hudson, PaperLink: a technique for hyperlinking from real paper to electronic content, Proceedings of the SIGCHI conference on Human factors in computing systems, p.327-334, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258782]
• 5: Daniel Avrahami , Scott E. Hudson , Thomas P. Moran , Brian D. Williams, Guided gesture support in the paper PDA, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502385]
• 6: Brian P. Bailey , Joseph A. Konstan, Are informal tools better?: comparing DEMAIS, pencil and paper, and authorware for early multimedia design, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642666]
• 7: Brian P. Bailey , Joseph A. Konstan , John V. Carlis, DEMAIS: designing multimedia applications with interactive storyboards, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500179]
• 8: Antonio Carzaniga , David S. Rosenblum , Alexander L. Wolf, Design and evaluation of a wide-area event notification service, ACM Transactions on Computer Systems (TOCS), v.19 n.3, p.332-383, Aug. 2001 [doi>10.1145/380749.380767]
• 9: Philip R. Cohen , David R. McGee, Tangible multimodal interfaces for safety-critical applications, Communications of the ACM, v.47 n.1, January 2004 [doi>10.1145/962081.962103]
• 10: Cunha, P. C. 2004. The I-Book: A freely interpretive notebook for biologists. In Proceedings of Participatory Design Conference'04. pp. 159--162.
• 11: Marc Dymetman , Max Copperman, Intelligent Paper, Proceedings of the 7th International Conference on Electronic Publishing, Held Jointly with the 4th International Conference on Raster Imaging and Digital Typography: Electronic Publishing, Artistic Imaging, and Digital Typography, p.392-406, March 30-April 03, 1998
• 12: Ebrahimi, R. 2005. Reading V2.0 : Next generation reading Systems and the PENHance Plug-in, unpublished honors thesis, University of California, San Diego, CA.
• 13: Google. 2005. Google Books, http://books.google.com.
• 14: François Guimbretière, Paper augmented digital documents, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.51-60, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964702]
• 15: François Guimbretière , Maureen Stone , Terry Winograd, Fluid interaction with high-resolution wall-size displays, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502353]
• 16: Gary Hardock , Gordon Kurtenbach , William Buxton, A marking based interface for collaborative writing, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.259-266, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168669]
• 17: Jeremy M. Heiner , Scott E. Hudson , Kenichiro Tanaka, Linking and messaging from real paper in the Paper PDA, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.179-186, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322600]
• 18: Ken Hinckley , Patrick Baudisch , Gonzalo Ramos , Francois Guimbretiere, Design and analysis of delimiters for selection-action pen gesture phrases in scriboli, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055035]
• 19: Ken Hinckley , Gonzalo Ramos , Francois Guimbretiere , Patrick Baudisch , Marc Smith, Stitching: pen gestures that span multiple displays, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989866]
• 20: IBM. 1998. CrossPadd, http://www.research.ibm.com/electricInk/.
• 21: Walter Johnson , Herbert Jellinek , Leigh Klotz, Jr. , Ramana Rao , Stuart K. Card, Bridging the paper and electronic worlds: the paper user interface, Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems, p.507-512, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169445]
• 22: Levent Burak Kara , Thomas F. Stahovich, Hierarchical parsing and recognition of hand-sketched diagrams, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029636]
• 23: Khan, F. 1993. A survey of note-taking practices. Tech Rep. HPL-93-107. Personal Systems Laboratory, HP Laboratory.
• 24: Scott R. Klemmer , Jamey Graham , Gregory J. Wolff , James A. Landay, Books with voices: paper transcripts as a physical interface to oral histories, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642628]
• 25: Scott R. Klemmer , Mark W. Newman , Ryan Farrell , Mark Bilezikjian , James A. Landay, The designers' outpost: a tangible interface for collaborative web site, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502350]
• 26: Hideki Koike , Yoichi Sato , Yoshinori Kobayashi , Hiroaki Tobita , Motoki Kobayashi, Interactive textbook and interactive Venn diagram: natural and intuitive interfaces on augmented desk system, Proceedings of the SIGCHI conference on Human factors in computing systems, p.121-128, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332415]
• 27: James A. Landay , Brad A. Myers, Interactive sketching for the early stages of user interface design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.43-50, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223910]
• 28: LeapFrog. 2005. Fly Pen, http://www.leapfrog.com.
• 29: Johnny C. Lee , Paul H. Dietz , Dan Maynes-Aminzade , Ramesh Raskar , Scott E. Hudson, Automatic projector calibration with embedded light sensors, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029653]
• 30: Yang Li , Ken Hinckley , Zhiwei Guan , James A. Landay, Experimental analysis of mode switching techniques in pen-based user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055036]
• 31: Chunyuan Liao , François Guimbretière , Ken Hinckley, PapierCraft: a command system for interactive paper, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095074]
• 32: Chunyuan Liao , François Guimbretière , Corinna E. Loeckenhoff, Pen-top feedback for paper-based interfaces, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166285]
• 33: James Lin , Mark W. Newman , Jason I. Hong , James A. Landay, DENIM: finding a tighter fit between tools and practice for Web site design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.510-517, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332486]
• 34: Logitech. 2004. IO Digital Pen, http://www.logitech.com.
• 35: W. Mackay , D. Pagani, Video mosaic: laying out time in a physical space, Proceedings of the second ACM international conference on Multimedia, p.165-172, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192646]
• 36: Wendy E. Mackay , Anne-Laure Fayard , Laurent Frobert , Lionel Médini, Reinventing the familiar: exploring an augmented reality design space for air traffic control, Proceedings of the SIGCHI conference on Human factors in computing systems, p.558-565, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274719]
• 37: W. E. Mackay , D. S. Pagani , L. Faber , B. Inwood , P. Launiainen , L. Brenta , V. Pouzol, Ariel: augmenting paper engineering drawings, Conference companion on Human factors in computing systems, p.421-422, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223763]
• 38: Wendy E. Mackay , Guillaume Pothier , Catherine Letondal , Kaare Bøegh , Hans Erik Sørensen, The missing link: augmenting biology laboratory notebooks, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.571992]
• 39: Mansouri-Samani, M. and Sloman, M. 1997. GEM: A generalised event monitoring language for distributed systems. IEE/IOP/BCS Distrib. Syst. Eng. J. 4, 25.
• 40: Catherine C. Marshall , Morgan N. Price , Gene Golovchinsky , Bill N. Schilit, Introducing a digital library reading appliance into a reading group, Proceedings of the fourth ACM conference on Digital libraries, p.77-84, August 11-14, 1999, Berkeley, California, United States [doi>10.1145/313238.313262]
• 41: Microsoft. 2003. OneNote, http://www.microsoft.com/onenote.
• 42: Microsoft. 2005. Microsoft Windows XP Tablet PC Edition 2005 Recognizer Pack.
• 43: Robert C. Miller , Brad A. Myers, Synchronizing clipboards of multiple computers, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.65-66, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322584]
• 44: Moira C. Norrie , Beat Signer, Switching Over to Paper: A New Web Channel, Proceedings of the Fourth International Conference on Web Information Systems Engineering, p.209, December 10-12, 2003
• 45: Kenton O'Hara , Abigail Sellen, A comparison of reading paper and on-line documents, Proceedings of the SIGCHI conference on Human factors in computing systems, p.335-342, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258787]
• 46: Dan R. Olsen, Jr. , Trent Taufer , Jerry Alan Fails, ScreenCrayons: annotating anything, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029663]
• 47: Morgan N. Price , Gene Golovchinsky , Bill N. Schilit, Linking by inking: trailblazing in a paper-like hypertext, Proceedings of the ninth ACM conference on Hypertext and hypermedia : links, objects, time and space---structure in hypermedia systems: links, objects, time and space---structure in hypermedia systems, p.30-39, June 20-24, 1998, Pittsburgh, Pennsylvania, United States [doi>10.1145/276627.276631]
• 48: Jun Rekimoto, Pick-and-drop: a direct manipulation technique for multiple computer environments, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.31-39, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263505]
• 49: Jun Rekimoto, Time-machine computing: a time-centric approach for the information environment, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.45-54, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322582]
• 50: Jun Rekimoto , Masanori Saitoh, Augmented surfaces: a spatially continuous work space for hybrid computing environments, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.378-385, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303113]
• 51: Eric Saund , David Fleet , Daniel Larner , James Mahoney, Perceptually-supported image editing of text and graphics, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.183-192, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964717]
• 52: Eric Saund , Edward Lank, Stylus input and editing without prior selection of mode, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.213-216, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964720]
• 53: Bill N. Schilit , Gene Golovchinsky , Morgan N. Price, Beyond paper: supporting active reading with free form digital ink annotations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.249-256, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274680]
• 54: Schön, D. A. 1983. The Reflective Practitioner: How Professionals Think in Action. Temple Smith, London, UK.
• 55: Schön, D. A. and Wiggins, G. 1992. Kinds of seeing and their functions in designing. Des. Stud. 13, 2, 135--156.
• 56: Abigail J. Sellen , Richard H.R. Harper, The Myth of the Paperless Office, MIT Press, Cambridge, MA, 2003
• 57: Sensiva. 2005. http://www.sensiva.com/.
• 58: Hyunyoung Song , François Guimbretière , Chang Hu , Hod Lipson, ModelCraft: capturing freehand annotations and edits on physical 3D models, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166258]
• 59: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 60: Lynn D. Wilcox , Bill N. Schilit , Nitin Sawhney, Dynomite: a dynamically organized ink and audio notebook, Proceedings of the SIGCHI conference on Human factors in computing systems, p.186-193, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258700]
• 61: Ron Yeh , Chunyuan Liao , Scott Klemmer , François Guimbretière , Brian Lee , Boyko Kakaradov , Jeannie Stamberger , Andreas Paepcke, ButterflyNet: a mobile capture and access system for field biology research, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124859]
• 62: Robert Zeleznik , Timothy Miller, Fluid inking: augmenting the medium of free-form inking with gestures, Proceedings of Graphics Interface 2006, June 07-09, 2006, Quebec, Canada
• 63: Shengdong Zhao , Ravin Balakrishnan, Simple vs. compound mark hierarchical marking menus, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029639]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Design, Documentation, Human Factors, Management

Keywords:
Paper interfaces, distributed systems, gesture-based interfaces, marking interfaces, paper-augmented digital documents, tablet computers, ubiquitous computing