Head gesture recognition in intelligent interfaces: the role of context in improving recognition

ABSTRACT

Acknowledging an interruption with a nod of the head is a natural and intuitive communication gesture which can be performed without significantly disturbing a primary interface activity. In this paper we describe vision-based head gesture recognition techniques and their use for common user interface commands. We explore two prototype perceptual interface components which use detected head gestures for dialog box confirmation and document browsing, respectively. Tracking is performed using stereo-based alignment, and recognition proceeds using a trained discriminative classifier. An additional context learning component is described, which exploits interface context to obtain robust performance. User studies with prototype recognition components indicate quantitative and qualitative benefits of gesture-based confirmation over conventional alternatives.

REFERENCES

• 1: Cynthia Breazeal , Lijin Aryananda, Recognition of Affective Communicative Intent in Robot-Directed Speech, Autonomous Robots, v.12 n.1, p.83-104, January 2002 [doi>10.1023/A:1013215010749]
• 2: J. Cassell , T. Bickmore , M. Billinghurst , L. Campbell , K. Chang , H. Vilhjálmsson , H. Yan, Embodiment in conversational interfaces: Rea, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.520-527, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303150]
• 3: Timothy Bickmore , Justine Cassell, Relational agents: a model and implementation of building user trust, Proceedings of the SIGCHI conference on Human factors in computing systems, p.396-403, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365304]
• 4: H. Clark and E. Schaefer. Contributing to discourse. Cognitive Science, 13:259--294, 1989.
• 5: C. J. Cohen , G. J. Beach , G. Foulk, A Basic Hand Gesture Control System for PC Applications, Proceedings of the 30th on Applied Imagery Pattern Recognition Workshop, p.74, October 10-12, 2001
• 6: James W. Davis , Serge Vaks, A perceptual user interface for recognizing head gesture acknowledgements, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971504]
• 7: S. Fujie, Y. Ejiri, K. Nakajima, Y. Matsusaka, and T. Kobayashi. A conversation robot using head gesture recognition as para-linguistic information. In Proceedings of 13th IEEE International Workshop on Robot and Human Communication, RO-MAN 2004, pages 159--164, September 2004.
• 8: Robert J. K. Jacob, Eye tracking in advanced interface design, Virtual environments and advanced interface design, Oxford University Press, Inc., New York, NY, 1995
• 9: Ashish Kapoor , Rosalind W. Picard, A real-time head nod and shake detector, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971509]
• 10: Shinjiro Kawato , Jun Ohya, Real-Time Detection of Nodding and Head-Shaking by Directly Detecting and Tracking the "Between-Eyes", Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.40, March 26-30, 2000
• 11: Rick Kjeldsen, Head Gestures for Computer Control, Proceedings of the IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems (RATFG-RTS'01), p.61, July 13-August 13, 2001
• 12: S. Lenman, L. Bretzer, and B. Thuresson. Computer vision based hand gesture interfaces for human-computer interaction. Technical Report CID-172, Center for User Oriented IT Design, June 2002.
• 13: Louis-Philippe Morency , Ali Rahimi , Neal Checka , Trevor Darrell, Fast Stereo-Based Head Tracking for Interactive Environments, Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, p.390, May 20-21, 2002
• 14: L.-P. Morency, A. Rahimi, and T. Darrell. Adaptive view-based appearance model. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, volume~1, pages 803--810, 2003.
• 15: Louis-Philippe Morency , Candace Sidner , Christopher Lee , Trevor Darrell, Contextual recognition of head gestures, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088470]
• 16: Yukiko I. Nakano , Gabe Reinstein , Tom Stocky , Justine Cassell, Towards a model of face-to-face grounding, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.553-561, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075166]
• 17: Charles Rich , Candace L. Sidner , Neal Lesh, Collagen: applying collaborative discourse theory to human-computer interaction, AI Magazine, v.22 n.4, p.15-25, Winter 2001
• 18: Candace L. Sidner , Cory D. Kidd , Christopher Lee , Neal Lesh, Where to look: a study of human-robot engagement, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964458]
• 19: Sidner, Lee, and N. Lesh. Engagement when looking: Behaviors for robots when collaborating with people. In Diabruck: Proceedings of the 7th workshop on the Semantic and Pragmatics of Dialogue, pages 123--130, University of Saarland, 2003. I. Kruiff-Korbayova and C. Kosny (eds.).
• 20: K. Toyama. Look, ma - no hands!hands-free cursor control with real-time 3D face tracking. In PUI98, 1998.
• 21: Wikipedia. Wikipedia encyclopedia. http://en.wikipedia.org/wiki/Dialog_box.
• 22: Shumin Zhai , Carlos Morimoto , Steven Ihde, Manual and gaze input cascaded (MAGIC) pointing, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.246-253, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303053]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Algorithms, Design, Experimentation, Human Factors, Performance

Keywords:
IUI design, context-based recognition, head gesture, multi-modal input, nod recognition, nodding, user study