PenLight: combining a mobile projector and a digital pen for dynamic visual overlay

ABSTRACT

Digital pen systems, originally designed to digitize annotations made on physical paper, are evolving to permit a wider variety of applications. Although the type and quality of pen feedback (e.g., haptic, audio, and visual) have a huge impact on advancing the digital pen technology, dynamic visual feedback has yet to be fully investigated. In parallel, miniature projectors are an emerging technology with the potential to enhance visual feedback for small mobile computing devices. In this paper we present the PenLight system, which is a testbed to explore the interaction design space and its accompanying interaction techniques in a digital pen embedded with a spatially-aware miniature projector. Using our prototype, that simulates a miniature projection (via a standard video projector), we visually augment paper documents, giving the user immediate access to additional information and computational tools. We also show how virtual ink can be managed in single and multi-user environments to aid collaboration and data management. User evaluation with professional architects indicated promise of our proposed techniques and their potential utility in the paper-intensive domain of architecture.

REFERENCES

• 1: Anoto, Development Guide for Service Enabled by Anoto Functionality. 2002, Anoto.
• 2: Georg Apitz , François Guimbretière, CrossY: a crossing-based drawing application, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029635]
• 3: Toshifumi Arai , Dietmar Aust , Scott E. Hudson, PaperLink: a technique for hyperlinking from real paper to electronic content, Proceedings of the SIGCHI conference on Human factors in computing systems, p.327-334, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258782]
• 4: James Arvo , Kevin Novins, Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.73-80, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354413]
• 5: Patrick Baudisch , Ruth Rosenholtz, Halo: a technique for visualizing off-screen objects, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642695]
• 6: Eric A. Bier , Maureen C. Stone , Ken Pier , William Buxton , Tony D. DeRose, Toolglass and magic lenses: the see-through interface, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.73-80, August 02-06, 1993, Anaheim, CA [doi>10.1145/166117.166126]
• 7: Xiang Cao , Ravin Balakrishnan, Interacting with dynamically defined information spaces using a handheld projector and a pen, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166289]
• 8: Xiang Cao , Clifton Forlines , Ravin Balakrishnan, Multi-user interaction using handheld projectors, Proceedings of the 20th annual ACM symposium on User interface software and technology, October 07-10, 2007, Newport, Rhode Island, USA [doi>10.1145/1294211.1294220]
• 9: Eastman, C., P. Teicholz, R. Sacks, and K. Liston, BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers and Contractors. 2008: Wiley.
• 10: Erwin, D.E., Further Evidence for Two Components in Visual Persistence. Journal of Experimental Psychology: Human Perception and Performance, 2, 2, 191--209, 1976.
• 11: George W. Fitzmaurice, Situated information spaces and spatially aware palmtop computers, Communications of the ACM, v.36 n.7, p.39-49, July 1993 [doi>10.1145/159544.159566]
• 12: Matthew Flagg , James M. Rehg, Projector-guided painting, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166290]
• 13: Flypentop, Flypentop Computer http://www.flypentop.com/.
• 14: Clifton Forlines , Ravin Balakrishnan , Paul Beardsley , Jeroen van Baar , Ramesh Raskar, Zoom-and-pick: facilitating visual zooming and precision pointing with interactive handheld projectors, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095046]
• 15: Tovi Grossman , Ken Hinckley , Patrick Baudisch , Maneesh Agrawala , Ravin Balakrishnan, Hover widgets: using the tracking state to extend the capabilities of pen-operated devices, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124898]
• 16: François Guimbretière, Paper augmented digital documents, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.51-60, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964702]
• 17: Janne Heikkila , Olli Silven, A Four-step Camera Calibration Procedure with Implicit Image Correction, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.1106, June 17-19, 1997
• 18: Jeremy M. Heiner , Scott E. Hudson , Kenichiro Tanaka, Linking and messaging from real paper in the Paper PDA, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.179-186, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322600]
• 19: Ken Hinckley , Randy Pausch , John C. Goble , Neal F. Kassell, A survey of design issues in spatial input, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.213-222, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192501]
• 20: David Holman , Roel Vertegaal , Mark Altosaar , Nikolaus Troje , Derek Johns, Paper windows: interaction techniques for digital paper, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055054]
• 21: Keisuke Tateno , Itaru Kitahara , Yuichi Ohta, A nested marker for augmented reality, ACM SIGGRAPH 2006 Sketches, July 30-August 03, 2006, Boston, Massachusetts [doi>10.1145/1179849.1180039]
• 22: Johnny C. Lee , Paul H. Dietz , Darren Leigh , William S. Yerazunis , Scott E. Hudson, Haptic pen: a tactile feedback stylus for touch screens, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029682]
• 23: Chunyuan Liao , François Guimbretière , Corinna E. Loeckenhoff, Pen-top feedback for paper-based interfaces, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166285]
• 24: Chunyuan Liao , François Guimbretière , Ken Hinckley , Jim Hollan, Papiercraft: A gesture-based command system for interactive paper, ACM Transactions on Computer-Human Interaction (TOCHI), v.14 n.4, p.1-27, January 2008 [doi>10.1145/1314683.1314686]
• 25: LiveScribe, http://www.livescribe.com/
• 26: Light Blue Optics http://www.lightblueoptics.com/
• 27: W. E. Mackay , D. S. Pagani , L. Faber , B. Inwood , P. Launiainen , L. Brenta , V. Pouzol, Ariel: augmenting paper engineering drawings, Conference companion on Human factors in computing systems, p.421-422, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223355.223763]
• 28: Wendy E. Mackay , Guillaume Pothier , Catherine Letondal , Kaare Bøegh , Hans Erik Sørensen, The missing link: augmenting biology laboratory notebooks, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.571992]
• 29: Norman, D., Psychology of everyday things. 1988: Basic Books.
• 30: Ken Perlin , Jon Meyer, Nested user interface components, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.11-18, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322578]
• 31: Thomas A. Phelps , Robert Wilensky, Multivalent Annotations, Proceedings of the First European Conference on Research and Advanced Technology for Digital Libraries, p.287-303, September 01-03, 1997
• 32: Beat Signer , Moira C. Norrie, PaperPoint: a paper-based presentation and interactive paper prototyping tool, Proceedings of the 1st international conference on Tangible and embedded interaction, February 15-17, 2007, Baton Rouge, Louisiana [doi>10.1145/1226969.1226981]
• 33: Sriram Subramanian , Dzimitry Aliakseyeu , Andrés Lucero, Multi-layer interaction for digital tables, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166295]
• 34: Michael Tsang , George W. Fitzmaurice , Gordon Kurtenbach , Azam Khan , Bill Buxton, Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572001]
• 35: MicroVision http://www.microvision.com/
• 36: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 37: Ka-Ping Yee, Peephole displays: pen interaction on spatially aware handheld computers, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642613]
• 38: Ron Yeh , Chunyuan Liao , Scott Klemmer , François Guimbretière , Brian Lee , Boyko Kakaradov , Jeannie Stamberger , Andreas Paepcke, ButterflyNet: a mobile capture and access system for field biology research, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124859]
• 39: Shumin Zhai , Stéphane Conversy , Michel Beaudouin-Lafon , Yves Guiard, Human on-line response to target expansion, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642644]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Design, Human Factors

Keywords:
digital pen input, mobile projector, multi-layer interaction, spatially-aware display