A real-time head nod and shake detector

ABSTRACT

Head nods and head shakes are non-verbal gestures used often to communicate intent, emotion and to perform conversational functions. We describe a vision-based system that detects head nods and head shakes in real time and can act as a useful and basic interface to a machine. We use an infrared sensitive camera equipped with infrared LEDs to track pupils. The directions of head movements, determined using the position of pupils, are used as observations by a discrete Hidden Markov Model (HMM) based pattern analyzer to detect when a head nod/shake occurs. The system is trained and tested on natural data from ten users gathered in the presence of varied lighting and varied facial expressions. The system as described achieves a real time recognition accuracy of 78.46% on the test dataset.

REFERENCES

• 1: Justine Cassell, Nudge nudge wink wink: elements of face-to-face conversation for embodied conversational agents, Embodied conversational agents, MIT Press, Cambridge, MA, 2001
• 2: Q. Chen , H. Wu , T. Fukumoto , M. Yachida, 3D Head Pose Estimation without Feature Tracking, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.88, April 14-16, 1998
• 3: Darwin, Charles (1872). The Expression of the Emotions in Man and Animals, third edition. New York, Oxford University Press, 1998.
• 4: Givens D. B. Dictionary of gestures, signs & body language cues. http://members.aol.com/nonverbal2/diction1.htm# The NONVERBAL DICTIONARY
• 5: Haro, A., Essa, I., and Flickner, M. Detecting and Tracking Eyes by Using their Physiological Properties, Dynamics and Appearance, in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2000.
• 6: K. Heinzmann , A. Zelinsky, 3-D Facial Pose and Gaze Point Estimation Using a Robust Real-Time Tracking Paradigm, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.142, April 14-16, 1998
• 7: Shinjiro Kawato , Jun Ohya, Real-Time Detection of Nodding and Head-Shaking by Directly Detecting and Tracking the "Between-Eyes", Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.40, March 26-30, 2000
• 8: Morris, D. Bodytalk: The Meaning of Human Gestures. Crown Publishers, New York 1994.
• 9: Morimoto, C., Koons, D., Amir, A., Flickner, M. Pupil Detection and Tracking Using Multiple Light Sources. Technical Report, IBM Almaden Research Center, 1998. http://domino.watson.ibm.com/library/cyberdig.nsf/Home.
• 10: Rabiner, L. A tutorial on Hidden Markov Models and selected applications in speech recognition, in Proceedings of IEEE, volume 77, number 2, February 1989, 257--286.
• 11: Henry A. Rowley , Shumeet Baluja , Takeo Kanade, Neural Network-Based Face Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.1, p.23-38, January 1998 [doi>10.1109/34.655647]
• 12: H. A. Rowley , S. Baluja , T. Kanade, Rotation Invariant Neural Network-Based Face Detection, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.963, June 23-25, 1998
• 13: Ying-li Tian , Takeo Kanade , Jeffrey F. Cohn, Dual-State Parametric Eye Tracking, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.110, March 26-30, 2000
• 14: Alan L. Yuille , Peter W. Hallinan , David S. Cohen, Feature extraction from faces using deformable templates, International Journal of Computer Vision, v.8 n.2, p.99-111, Aug. 1992 [doi>10.1007/BF00127169]

INDEX TERMS

Keywords:
HMM, head nod, head shake, pupil tracking