Interacting with spatially augmented reality

ABSTRACT

We present the notion of projector-based spatially augmented reality (SAR), and explore how it can be used as an effective user interface to enable users to easily and naturally interact with their real physical environment and the virtual environment. In SAR, the user's physical environment is illuminated with images projected from the projectors. We then describe a framework that can easily incorporate different types of interactions on a continuum of display surfaces and input devices. More specifically, in the framework, we first establish an understanding of the 3D relationship between the user, the projectors and the display surfaces. Then, with the help of some new calibration and rendering techniques, only a simple procedure is needed to effectively illuminate the display surfaces. We describe various underlying techniques and discuss the results in the context of three different applications.

REFERENCES

• 1: Ronald Azuma , Gary Bishop, Improving static and dynamic registration in an optical see-through HMD, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.197-204, July 1994 [doi>10.1145/192161.192199]
• 2: Deepak Bandyopadhyay , Ramesh Raskar , Andrei State , Henry Fuchs, Dynamic Spatially Augmented 3D Painting, University of North Carolina at Chapel Hill, Chapel Hill, NC, 2001
• 3: {Bennett98} David T. Bennett. Chairman and Co-Founder of Alternate Realities Corporation, 215 Southport Drive Suite 1300, Morrisville, NC 27560, USA.
• 4: {Bryson97} Bryson, Steve, David Zeltzer, Mark T. Bolas, Bertrand de La Chapelle, and David Bennett. The Future of Virtual Reality: Head Mounted Displays Versus Spatially Immersive Displays, SIGGRAPH 97 (Panel) Conference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley, pp. 485-486, August 1997.
• 5: Carolina Cruz-Neira , Daniel J. Sandin , Thomas A. DeFanti, Surround-screen projection-based virtual reality: the design and implementation of the CAVE, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.135-142, September 1993 [doi>10.1145/166117.166134]
• 6: Julie O'B. Dorsey , Françis X. Sillion , Donald P. Greenberg, Design and simulation of opera lighting and projection effects, Proceedings of the 18th annual conference on Computer graphics and interactive techniques, p.41-50, July 1991
• 7: Olivier Faugeras, Three-dimensional computer vision: a geometric viewpoint, MIT Press, Cambridge, MA, 1993
• 8: Ken Hinckley , Mike Sinclair , Erik Hanson , Richard Szeliski , Matt Conway, The VideoMouse: a camera-based multi-degree-of-freedom input device, Proceedings of the 12th annual ACM symposium on User interface software and technology, p.103-112, November 07-10, 1999, Asheville, North Carolina, United States [doi>10.1145/320719.322591]
• 9: Richard Lee Holloway, Registration errors in augmented reality systems, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1996
• 10: {Hologlobe} (Cited July 10, 1998) http://www.3dmedia.com
• 11: {Howard00} HowardModels.com, 7944 Central Avenue, Toledo, OH 43617. http://www.howardweb.com/model/
• 12: {Ishii96} Ishii, Hiroshi & Ulmer, B. Tangible Bits: Towards seamless interfaces between people, bits and atoms. Proceedings of CHI'96. New York: ACM.
• 13: {Jarvis97} Kevin Jarvis, Real Time 60Hz Distortion Correction on a Silicon Graphics IG. In Real Time Graphics, Vol. 5, No. 7, pp. 6-7, February 1997.
• 14: {Liljegren90} G. E. Liljegren and E. L. Foster. Figure with Back Projected Image Using Fiber Optics. US Patent # 4,978.216, Walt Disney Company, USA, December 1990.
• 15: Kok-Lim Low , Greg Welch , Anselmo Lastra , Henry Fuchs, Life-sized projector-based dioramas, Proceedings of the ACM symposium on Virtual reality software and technology, November 15-17, 2001, Baniff, Alberta, Canada [doi>10.1145/505008.505026]
• 16: {Mankoff98} Jennifer, Jonathan Somers and Gregory D. Abowd. Bringing People and Places Together with Dual Augmentation. Collaborative Virtual Environments, Manchester. June 1998.
• 17: {Max91} Nelson Max. Computer animation of photosynthesis. Proceedings of the Second Eurographics Workshop on Animation and Simulation, Vienna, pp. 25-39, 1991.
• 18: {Milgram94a} P Milgram and F Kishino. A taxonomy of mixed reality visual displays. IEICE (Institute of Electronics, Information and Communication Engineers) Transactions on Information and Systems, Special issue on Networked Reality, Dec 1994.
• 19: {Milgram94b} P Milgram, H Takemura, A Utsumi and F Kishino. Augmented Reality: A class of displays on the reality-virtuality continuum. SPIE Vol. 2351-34, Telemanipulator and Telepresence Technologies, 1994.
• 20: Elizabeth Mynatt , Douglas Blattner , Meera M. Blattner , Blair MacIntyre , Jennifer Mankoff, Augmenting home and office environments, Proceedings of the third international ACM conference on Assistive technologies, p.169-172, April 15-17, 1998, Marina del Rey, California, United States [doi>10.1145/274497.274529]
• 21: Elizabeth D. Mynatt , Takeo Igarashi , W. Keith Edwards , Anthony LaMarca, Flatland: new dimensions in office whiteboards, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.346-353, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303108]
• 22: {Naimark84} M. Naimark. Displacements. An exhibit at the San Francisco Museum of Modern Art, San Francisco, CA (USA), 1984.
• 23: Dan R. Olsen, Jr. , Travis Nielsen, Laser pointer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.17-22, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365030]
• 24: Ramesh Raskar , Greg Welch , Matt Cutts , Adam Lake , Lev Stesin , Henry Fuchs, The office of the future: a unified approach to image-based modeling and spatially immersive displays, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.179-188, July 1998 [doi>10.1145/280814.280861]
• 25: Ramesh Raskar , Michael S. Brown , Ruigang Yang , Wei-Chao Chen , Greg Welch , Herman Towles , Brent Seales , Henry Fuchs, Multi-projector displays using camera-based registration, Proceedings of the conference on Visualization '99: celebrating ten years, p.161-168, October 1999, San Francisco, California, United States
• 26: Ramesh Raskar , Greg Welch , Kok-Lim Low , Deepak Bandyopadhyay, Shader Lamps: Animating Real Objects With Image-Based Illumination, Proceedings of the 12th Eurographics Workshop on Rendering Techniques, p.89-102, June 25-27, 2001
• 27: {Siio95} Siio, I. A Pointing Device for Virtual Desktop System. Proceedings of HCI International 1995, pp. 261-264, Elsevier Science, July 1995.
• 28: Andrei State , Gentaro Hirota , David T. Chen , William F. Garrett , Mark A. Livingston, Superior augmented reality registration by integrating landmark tracking and magnetic tracking, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.429-438, August 1996 [doi>10.1145/237170.237282]
• 29: Mark Segal , Carl Korobkin , Rolf van Widenfelt , Jim Foran , Paul Haeberli, Fast shadows and lighting effects using texture mapping, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.249-252, July 1992
• 30: {UnderKoffler97} John Underkoffler. A View From the Luminous Room. Springer-Verlag London Ltd., Personal Technologies (1997) 1:49-59.
• 31: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

General Terms:
Design