Pointing gesture recognition based on 3D-tracking of face, hands and head orientation

ABSTRACT

In this paper, we present a system capable of visually detecting pointing gestures and estimating the 3D pointing direction in real-time. In order to acquire input features for gesture recognition, we track the positions of a person's face and hands on image sequences provided by a stereo-camera. Hidden Markov Models (HMMs), trained on different phases of sample pointing gestures, are used to classify the 3D-trajectories in order to detect the occurrence of a gesture. When analyzing sample pointing gestures, we noticed that humans tend to look at the pointing target while performing the gesture. In order to utilize this behavior, we additionally measured head orientation by means of a magnetic sensor in a similar scenario. By using head orientation as an additional feature, we observed significant gains in both recall and precision of pointing gestures. Moreover, the percentage of correctly identified pointing targets improved significantly from 65% to 83%. For estimating the pointing direction, we comparatively used three approaches: 1) The line of sight between head and hand, 2) the forearm orientation, and 3) the head orientation.

REFERENCES

• 1: Paul P. Maglio , Teenie Matlock , Christopher S. Campbell , Shumin Zhai , Barton A. Smith, Gaze and Speech in Attentive User Interfaces, Proceedings of the Third International Conference on Advances in Multimodal Interfaces, p.1-7, October 14-16, 2000
• 2: B. Brumitt, J. Krumm, B. Meyers, and S. Shafer. Let There Be Light: Comparing Interfaces for Homes of the Future. IEEE Personal Communications, August 2000.
• 3: Christopher Richard Wren , Ali Azarbayejani , Trevor Darrell , Alex Paul Pentland, Pfinder: Real-Time Tracking of the Human Body, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.780-785, July 1997 [doi>10.1109/34.598236]
• 4: A. Azarbayejani , A. Pentland, Real-Time Self-Calibrating Stereo Person Tracking Using 3-D Shape Estimation from Blob Features, Proceedings of the International Conference on Pattern Recognition (ICPR '96) Volume III-Volume 7276, p.627, August 25-29, 1996
• 5: T. Darrell , G. Gordon , M. Harville , J. Woodfill, Integrated Person Tracking Using Stereo, Color, and Pattern Detection, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.601, June 23-25, 1998
• 6: T. Starner and A. Pentland. Visual Recognition of American Sign Language Using Hidden Markov Models. M.I.T. Media Laboratory, Perceptual Computing Section, Cambridge MA, USA, 1994.
• 7: D. A. Becker. Sensei: A Real-Time Recognition, Feedback and Training System for T'ai Chi Gestures. M.I.T. Media Lab Perceptual Computing Group Technical Report No. 426, 1997.
• 8: Andrew D. Wilson , Aaron F. Bobick, Recognition and Interpretation of Parametric Gesture, Proceedings of the Sixth International Conference on Computer Vision, p.329, January 04-07, 1998
• 9: I. Poddar, Y. Sethi, E. Ozyildiz, and R. Sharma. Toward Natural Gesture/Speech HCI: A Case Study of Weather Narration. Proc. Workshop on Perceptual User Interfaces (PUI98), San Francisco, USA. 1998.
• 10: R. E. Kahn , M. J. Swain , P. N. Prokopowicz , R. J. Firby, Gesture recognition using the Perseus architecture, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.734, June 18-20, 1996
• 11: Nebojsa Jojic , Thomas Huang , Barry Brumitt , Brian Meyers , Steve Harris, Detection and Estimation of Pointing Gestures in Dense Disparity Maps, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.468, March 26-30, 2000
• 12: K. Konolige. Small Vision Systems: Hardware and Implementation. Eighth International Symposium on Robotics Research, Hayama, Japan, 1997.
• 13: J. Yang, W. Lu, and A. Waibel. Skin-color modeling and adaption. Technical Report of School of Computer Science, CMU, CMU-CS-97-146, 1997.
• 14: L. R. Rabiner. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proc. IEEE, 77 (2), 257--286, 1989.
• 15: L. W. Campbell , D. A. Becker , A. Azarbayejani , A. F. Bobick , A. Pentland, Invariant features for 3-D gesture recognition, Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96), p.157, October 14-16, 1996

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
computer vision, gesture recognition, person tracking, pointing gestures