Real-time hand-tracking with a color glove

ABSTRACT

Articulated hand-tracking systems have been widely used in virtual reality but are rarely deployed in consumer applications due to their price and complexity. In this paper, we propose an easy-to-use and inexpensive system that facilitates 3-D articulated user-input using the hands. Our approach uses a single camera to track a hand wearing an ordinary cloth glove that is imprinted with a custom pattern. The pattern is designed to simplify the pose estimation problem, allowing us to employ a nearest-neighbor approach to track hands at interactive rates. We describe several proof-of-concept applications enabled by our system that we hope will provide a foundation for new interactions in modeling, animation control and augmented reality.

REFERENCES

• 1: Maneesh Agrawala , Andrew C. Beers , Ian McDowall , Bernd Fröhlich , Mark Bolas , Pat Hanrahan, The two-user Responsive Workbench: support for collaboration through individual views of a shared space, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.327-332, August 1997 [doi>10.1145/258734.258875]
• 2: Athitsos, V., and Sclaroff, S. 2003. Estimating 3D hand pose from a cluttered image. In Computer Vision and Pattern Recognition (CVPR), vol. 2, 432--439.
• 3: Athitsos, V., Alon, J., Sclaroff, S., and Kollios, G. 2004. BoostMap: A method for efficient approximate similarity rankings. In Computer Vision and Pattern Recognition (CVPR), vol. 2, 268--275.
• 4: Connelly Barnes , David E. Jacobs , Jason Sanders , Dan B Goldman , Szymon Rusinkiewicz , Adam Finkelstein , Maneesh Agrawala, Video puppetry: a performative interface for cutout animation, ACM Transactions on Graphics (TOG), v.27 n.5, December 2008
• 5: Hrvoje Benko , Edward W. Ishak, Cross-Dimensional Gestural Interaction Techniques for Hybrid Imrnersive Environments, Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality, p.209-216, 327, March 12-16, 2005 [doi>10.1109/VR.2005.27]
• 6: Hamilton Y. Chong , Steven J. Gortler , Todd Zickler, A perception-based color space for illumination-invariant image processing, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 7: de La Gorce, M., Paragios, N., and Fleet, D. 2008. Model-Based Hand Tracking with Texture, Shading and Self-occlusions. In Computer Vision and Pattern Recognition (CVPR), 1--8.
• 8: Dewaele, G., Devernay, F., Horaud, R. P., and Forbes, F. 2006. The alignment between 3-d data and articulated shapes with bending surfaces. In European Conference on Computer Vision (ECCV), 578--591.
• 9: Pushkar Dhawale , Masood Masoodian , Bill Rogers, Bare-hand 3D gesture input to interactive systems, Proceedings of the 7th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: design centered HCI, p.25-32, July 06-07, 2006, Christchurch, New Zealand [doi>10.1145/1152760.1152764]
• 10: Mira Dontcheva , Gary Yngve , Zoran Popović, Layered acting for character animation, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 11: Dorner, B. 1994. Chasing the Colour Glove: Visual Hand Tracking. Master's thesis, Simon Fraser University.
• 12: Tovi Grossman , Daniel Wigdor , Ravin Balakrishnan, Multi-finger gestural interaction with 3d volumetric displays, Proceedings of the 17th annual ACM symposium on User interface software and technology, October 24-27, 2004, Santa Fe, NM, USA [doi>10.1145/1029632.1029644]
• 13: Igor Guskov , Sergey Klibanov , Benjamin Bryant, Trackable surfaces, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 14: Daniel F. Keefe , David B. Karelitz , Eileen L. Vote , David H. Laidlaw, Artistic Collaboration in Designing VR Visualizations, IEEE Computer Graphics and Applications, v.25 n.2, p.18-23, March 2005 [doi>10.1109/MCG.2005.34]
• 15: Kersten, D., Mamassian, P., and Knill, D. 1997. Moving cast shadows induce apparent motion in depth. Perception 26, 2, 171--192.
• 16: Paul G. Kry , Dinesh K. Pai, Interaction capture and synthesis, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 17: Paul G. Kry , Adeline Pihuit , Adrien Bernhardt , Marie-Paule Cani, HandNavigator: hands-on interaction for desktop virtual reality, Proceedings of the 2008 ACM symposium on Virtual reality software and technology, October 27-29, 2008, Bordeaux, France [doi>10.1145/1450579.1450591]
• 18: Wai-Chun Lam , Feng Zou , Taku Komura, Motion editing with data glove, Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology, p.337-342, June 03-05, 2005, Singapore [doi>10.1145/1067343.1067393]
• 19: ChanSu Lee , SangWon Ghyme , ChanJong Park , KwangYun Wohn, The control of avatar motion using hand gesture, Proceedings of the ACM symposium on Virtual reality software and technology, p.59-65, November 02-05, 1998, Taipei, Taiwan [doi>10.1145/293701.293709]
• 20: Ying Li , Jiaxin L. Fu , Nancy S. Pollard, Data-Driven Grasp Synthesis Using Shape Matching and Task-Based Pruning, IEEE Transactions on Visualization and Computer Graphics, v.13 n.4, p.732-747, July 2007 [doi>10.1109/TVCG.2007.1033]
• 21: Alex Olwal , Hrvoje Benko , Steven Feiner, SenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System, Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.300, October 07-10, 2003
• 22: Jun Park , Yeo-Lip Yoon, LED-Glove Based Interactions in Multi-Modal Displays for Teleconferencing, Proceedings of the 16th International Conference on Artificial Reality and Telexistence--Workshops, p.395-399, November 29-December 01, 2006 [doi>10.1109/ICAT.2006.80]
• 23: Nancy S. Pollard , Victor Brian Zordan, Physically based grasping control from example, Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 29-31, 2005, Los Angeles, California [doi>10.1145/1073368.1073413]
• 24: Liu Ren , Gregory Shakhnarovich , Jessica K. Hodgins , Hanspeter Pfister , Paul Viola, Learning silhouette features for control of human motion, ACM Transactions on Graphics (TOG), v.24 n.4, p.1303-1331, October 2005 [doi>10.1145/1095878.1095882]
• 25: Markus Schlattman , Reinhard Klein, Simultaneous 4 gestures 6 DOF real-time two-hand tracking without any markers, Proceedings of the 2007 ACM symposium on Virtual reality software and technology, November 05-07, 2007, Newport Beach, California [doi>10.1145/1315184.1315188]
• 26: Scholz, V., Stich, T., Keckeisen, M., Wacker, M., and Magnor, M. A. 2005. Garment motion capture using color-coded patterns. Computer Graphics Forum 24, 3, 439--447.
• 27: Gregory Shakhnarovich , Paul Viola , Trevor Darrell, Fast Pose Estimation with Parameter-Sensitive Hashing, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.750, October 13-16, 2003
• 28: Jia Sheng , Ravin Balakrishnan , Karan Singh, An interface for virtual 3D sculpting via physical proxy, Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, November 29-December 02, 2006, Kuala Lumpur, Malaysia [doi>10.1145/1174429.1174467]
• 29: Takaaki Shiratori , Jessica K. Hodgins, Accelerometer-based user interfaces for the control of a physically simulated character, ACM Transactions on Graphics (TOG), v.27 n.5, December 2008
• 30: Thad Starner , Alex Pentland , Joshua Weaver, Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.12, p.1371-1375, December 1998 [doi>10.1109/34.735811]
• 31: Bjorn Stenger , Arasanathan Thayananthan , Philip H. S. Torr , Roberto Cipolla, Model-Based Hand Tracking Using a Hierarchical Bayesian Filter, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.9, p.1372-1384, September 2006 [doi>10.1109/TPAMI.2006.189]
• 32: David J. Sturman , David Zeltzer, A design method for “whole-hand” human-computer interaction, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.219-238, July 1993 [doi>10.1145/159161.159159]
• 33: Sudderth, E. B., Mandel, M. I., Freeman, T., and Willsky, S. 2004. Distributed occlusion reasoning for tracking with nonparametric belief propagation. In Neural Information Processing Systems (NIPS), 1369--1376.
• 34: Christian Theobalt , Irene Albrecht , Jörg Haber , Marcus Magnor , Hans-Peter Seidel, Pitching a baseball: tracking high-speed motion with multi-exposure images, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 35: Torralba, A., Fergus, R., and Freeman, W. T. 2007. Tiny images. Tech. Rep. MIT-CSAIL-TR-2007-024, Computer Science and Artificial Intelligence Lab, MIT.
• 36: Torralba, A., Fergus, R., and Weiss., Y. 2008. Small codes and large databases for recognition. In Computer Vision and Pattern Recognition (CVPR), vol. 2, 1--8.
• 37: Gerold Wesche , Hans-Peter Seidel, FreeDrawer: a free-form sketching system on the responsive workbench, Proceedings of the ACM symposium on Virtual reality software and technology, November 15-17, 2001, Baniff, Alberta, Canada [doi>10.1145/505008.505041]
• 38: White, R., and Forsyth, D. 2005. Deforming objects provide better camera calibration. Tech. Rep. UCB/EECS-2005-3, EECS Department, University of California, Berkeley.
• 39: Ryan White , Keenan Crane , D. A. Forsyth, Capturing and animating occluded cloth, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 40: Andrew D. Wilson , Shahram Izadi , Otmar Hilliges , Armando Garcia-Mendoza , David Kirk, Bringing physics to the surface, Proceedings of the 21st annual ACM symposium on User interface software and technology, October 19-22, 2008, Monterey, CA, USA [doi>10.1145/1449715.1449728]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Keywords:
augmented reality, hand tracking, motion capture, user interface