3D position, attitude and shape input using video tracking of hands and lips

ABSTRACT

Recent developments in video-tracking allow the outlines of moving, natural objects in a video-camera input stream to be tracked live, at full video-rate. Previous systems have been available to do this for specially illuminated objects or for naturally illuminated but polyhedral objects. Other systems have been able to track nonpolyhedral objects in motion, in some cases from live video, but following only centroids or key-points rather than tracking whole curves. The system described here can track accurately the curved silhouettes of moving non-polyhedral objects at frame-rate, for example hands, lips, legs, vehicles, fruit, and without any special hardware beyond a desktop workstation and a video-camera and framestore.The new algorithms are a synthesis of methods in deformable models, B-splines curve representation and control theory. This paper shows how such a facility can be used to turn parts of the body—for instance, hands and lips—into input devices. Rigid motion of a hand can be used as a 3D mouse with non-rigid gestures signalling a button press or the “lifting” of the mouse. Both rigid and non-rigid motions of lips can be tracked independently and used as inputs, for example to animate a computer-generated face.

REFERENCES

• 1: Karl J. Åström , Björn Wittenmark, Computer-controlled systems: theory and design (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1990
• 2: Karl Johan Astrom , Bjorn Wittenmark, Adaptive Control, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1994
• 3: A. Azarbayejani , T. Starner , B. Horowitz , A. Pentland, Visually Controlled Graphics, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.6, p.602-605, June 1993 [doi>10.1109/34.216730]
• 4: Y. Bar-Shalom, Tracking and data association, Academic Press Professional, Inc., San Diego, CA, 1987
• 5: Stephen Barnett. Matrices: Methods and Applications. OxfordUniversity Press, 1990.
• 6: Richard H. Bartels , John C. Beatty , Brian A. Barsky, An introduction to splines for use in computer graphics & geometric modeling, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1987
• 7: Andrew Blake , Rupert Curwen , Andrew Zisserman, A framework for spatiotemporal control in the tracking of visual contours, International Journal of Computer Vision, v.11 n.2, p.127-145, Oct. 1993 [doi>10.1007/BF01469225]
• 8: R. Cipolla and A. Blake. The dynamic analysis of apparent contours. In Proc. 3rd Int. Conf. on Computer Vision, pages 616-625, 1990.
• 9: T.F. Cootes, C.J. Taylor, A. Lanitis, D.H. Cooper, and J. Graham. Buiding and using flexible models incorporating grey-level information. In Proc. 4th Int. Conf. on Computer Vision, pages 242-246,1993.
• 10: Ernst Dieter Dickmanns , Volker Graefe, Applications of dynamic monocular machine vision, Machine Vision and Applications, v.1 n.4, p.241-261, 1988 [doi>10.1007/BF01212362]
• 11: I. D. Faux , M. J. Pratt, Computational Geometry for Design and Manufacture, Halsted Press, New York, NY, 1979
• 12: M. A. Fischler andR. A. Elschlager. The representationandmatchingof pictorial structures. IEEE. Trans. Computers, C-22(1), 1973.
• 13: Arthur Gelb, editor. Applied Optimal Estimation. MIT Press, Cambridge, MA, 1974.
• 14: U. Grenander , Y. Chow , D. M. Keenan, Hands: a pattern theoretic study of biological shapes, Springer-Verlag New York, Inc., New York, NY, 1991
• 15: Chris Harris, Tracking with rigid models, Active vision, MIT Press, Cambridge, MA, 1993
• 16: Berthold Klaus Paul Horn, Robot vision, MIT Press, Cambridge, MA, 1986
• 17: M. Kass, A. Witkin, and D. Terzopoulos. Snakes: active contour models. In Proc. 1st Int. Conf. on Computer Vision, pages 259-268,1987.
• 18: J.J. Koenderink and A.J. Van Doorn. Affine structure from motion. J. Optical Soc. of America A., 8(2):337-385,1991.
• 19: David G. Lowe, Robust model-based motion tracking through the integration of search and estimation, International Journal of Computer Vision, v.8 n.2, p.113-122, Aug. 1992 [doi>10.1007/BF00127170]
• 20: K. Mase and A. Pentland. Automatic lip-readingby optical flow analysis. Media Lab Report 117, MIT, 1991.
• 21: S. Menet, P. Saint-Marc, and G. Medioni. B-snakes: implementation and application to stereo. In Proceedings DARPA, pages 720-726, 1990.
• 22: Athanasios Papoulis, Probability and statistics, Prentice-Hall, Inc., Upper Saddle River, NJ, 1990
• 23: R. Szeliski and D. Terzopoulos. Physically-based and probabilistic modelingfor computer vision. In B. C. Vemuri, editor, Proc. SPIE 1570, Geometric Methods in Computer Vision, pages 140-152, San Diego, CA, July 1991. Society of Photo-Optical Instrumentation Engineers.
• 24: D. Terzopoulos and K. Waters. Physically-based facial modelling, analysis and animation. J. Visualization and COmputer Animation, 11(2), 1990.
• 25: Shimon Ullman , Ronen Basri, Recognition by Linear Combinations of Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.13 n.10, p.992-1006, October 1991 [doi>10.1109/34.99234]
• 26: C. Ware , D. R. Jessome, Using the bat: a six dimensional mouse for object placement, Proceedings on Graphics interface '88, p.119-124, December 1989, Edmonton, Alberta, Canada
• 27: Lance Williams, Performance-driven facial animation, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.235-242, September 1990, Dallas, TX, USA

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

General Terms:
Algorithms, Theory