Robust real-time upper body limb detection and tracking

ABSTRACT

We describe an efficient and robust system to detect and track the limbs of a human. Of special consideration in the design of this system are real-time and robustness issues. We thus utilize a detection/tracking scheme in which we detect the face and limbs of a user and then track the forearms of the found limbs. Detection occurs by first finding the face of a user. The location and color information from the face can then be used to find limbs. As skin color is a key visual feature in this system, we continuously search for faces and use them to update skin color information. Along with edge information, this is used in the subsequent forearm tracking. Robustness is implicit in this design, as the system automatically re-detects a limbs when its corresponding forearms is lost. This design is also conducive to real-time processing because while detection of the limbs can take up to seconds, tracking is on the order of milliseconds. Thus reasonable frame rates can be achieved with a short latency. Also, in this system we make use of multiple 2D limb tracking models to enhance tracking of the underlying 3D structure. This includes models for lateral forearm views (waving) as well as for pointing gestures. Experiments on test sequences demonstrate the efficacy of this approach.

REFERENCES

• 1: G. R. Bradski. The OpenCV Library. Dr. Dobb's Software Tools for the Professional Programmer, November 2000.
• 2: Cynthia Breazeal, Socially intelligent robots, interactions, v.12 n.2, March + April 2005 [doi>10.1145/1052438.1052455]
• 3: C. Bregler , J. Malik, Tracking People with Twists and Exponential Maps, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.8, June 23-25, 1998
• 4: Isaac Cohen , Hongxia Li, Inference of Human Postures by Classification of 3D Human Body Shape, Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, p.74, October 17-17, 2003
• 5: Dorin Comaniciu , Visvanathan Ramesh , Peter Meer, Kernel-Based Object Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.5, p.564-575, May 2003 [doi>10.1109/TPAMI.2003.1195991]
• 6: D. J. Feil-Seifer and M. J. Matarić. Socially assistive robotics. In Proceedings of the International Conference on Rehabilitation Robotics, Chicago, Il, July 2005.
• 7: Pedro F. Felzenszwalb , Daniel P. Huttenlocher, Pictorial Structures for Object Recognition, International Journal of Computer Vision, v.61 n.1, p.55-79, January 2005 [doi>10.1023/B:VISI.0000042934.15159.49]
• 8: T. Fong, I. Nourbakhsh, and K. Dautenhahn. A survey of socially interactive robots. Robotics and Autonomous Systems, Special issue on Socially Interactive Robots, 42(3-4):143--166, 2003.
• 9: A.R. François. Software architecture for computer vision. In G. Medioni and S. Kang, editors, Emerging Topics in Computer Vision, pages 585--654. Prentice Hall, 2004.
• 10: S. Ioffe , D. A. Forsyth, Probabilistic Methods for Finding People, International Journal of Computer Vision, v.43 n.1, p.45-68, June 2001 [doi>10.1023/A:1011179004708]
• 11: M. W. Lee and I. Cohen. Human upper body pose estimation in static images. In ECCV, pages II: 126--138, 2004.
• 12: David P. Miller, Assistive Robotics: An Overview, Assistive Technology and Artificial Intelligence, Applications in Robotics, User Interfaces and Natural Language Processing, p.126-136, January 1998
• 13: D. D. Morris , J. M. Rehg, Singularity Analysis for Articulated Object Tracking, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.289, June 23-25, 1998
• 14: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 15: D. Ramanan and D. A. Forsyth. Finding and tracking people from the bottom up. In CVPR, pages II: 467--474, 2003.
• 16: L. Sigal, S. Bhatia, S. Roth, M. J. Black, and M. Isard. Tracking loose-limbed people. In CVPR, pages I: 421--428, 2004.
• 17: C. Sminchisescu and B. Triggs. Covariance scaled sampling for monocular 3d body tracking. In CVPR, Kauai Marriott, Hawaii, December 2001.
• 18: Haihong Zhang , Weimin Huang , Zhiyong Huang , Liyuan Li, Affine Object Tracking with Kernel-Based Spatial-Color Representation, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.293-300, June 20-26, 2005 [doi>10.1109/CVPR.2005.65]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis; Motion

General Terms:
Human Factors, Performance, Reliability

Keywords:
body pose estimation, gesture and interaction analysis, limb detection, limb tracking, real-time vision