A realtime mixed reality system for seamless interaction between real and virtual objects

ABSTRACT

A mixed reality system where users interact with real and virtual 3D objects in a 3D world controlled by a physics engine is presented. To build the system we propose a hand detection and tracking system that maintains information about the way multiple users move their palms and fingers in 3D space using the feed from two fixed uncalibrated cameras. The complete 3D system can process and renders more than 30 frames per second.

REFERENCES

• 1: Bullet continuous collision detection and physics library. http://www.bulletphysics.com/Bullet/.
• 2: Microsoft surface. http://www.surface.com/.
• 3: Gunilla Borgefors, Distance transformations in digital images, Computer Vision, Graphics, and Image Processing, v.34 n.3, p.344-371, June 1986 [doi>10.1016/S0734-189X(86)80047-0]
• 4: G. Bradski. Computer vision face tracking for use in a perceptual user interface, 1998.
• 5: M. J. Jones and J. M. Rehg. Statistical color models with application to skin detection. In Computer Vision and Pattern Recognition (CVPR), pages 1274--1280, 1999.
• 6: D. Scharstein, R. Szeliski, and R. Zabih. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, 2001.
• 7: S. Suzuki and K. Abe. Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing, 30(1):32--46, April 1985.
• 8: C. H. Teh , R. T. Chin, On the Detection of Dominant Points on Digital Curves, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.11 n.8, p.859-872, August 1989 [doi>10.1109/34.31447]
• 9: G. Toussaint. Solving geometric problems with the rotating calipers, 1983.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms

Keywords:
hand segmentation, hand tracking, mixed reality systems