Visual and linguistic information in gesture classification

ABSTRACT

Classification of natural hand gestures is usually approached by applying pattern recognition to the movements of the hand. However, the gesture categories most frequently cited in the psychology literature are fundamentally multimodal; the definitions make reference to the surrounding linguistic context. We address the question of whether gestures are naturally multimodal, or whether they can be classified from hand-movement data alone. First, we describe an empirical study showing that the removal of auditory information significantly impairs the ability of human raters to classify gestures. Then we present an automatic gesture classification system based solely on an n-gram model of linguistic context; the system is intended to supplement a visual classifier, but achieves 66% accuracy on a three-class classification problem on its own. This represents higher accuracy than human raters achieve when presented with the same information.

REFERENCES

• 1: Jean Carletta, Assessing agreement on classification tasks: the kappa statistic, Computational Linguistics, v.22 n.2, June 1996
• 2: Cassell, J. A framework for gesture generation and interpretation. Computer Vision in Human-Machine Interaction. Cambridge University Press (1998), 191--215.
• 3: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 4: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 5: Eisenstein, J., and Davis, R. Natural gesture in descriptive monologues. UIST'03 Supplemental Proceedings. ACM Press (2003), 69--70.
• 6: Freeman, W. T., and Weissman, C. Television control by hand gestures. International Workshop on Automatic Face- and Gesture- Recognition. IEEE Press (1995), M. Bichsel, Ed., 179--183.
• 7: Ed Kaiser , Alex Olwal , David McGee , Hrvoje Benko , Andrea Corradini , Xiaoguang Li , Phil Cohen , Steven Feiner, Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958438]
• 8: Kendon, A. Conducting Interaction. Cambridge University Press, 1990.
• 9: Sanshzar Kettebekov , Rajeev Sharma, Toward Natural Gesture/Speech Control of a Large Display, Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, p.221-234, May 11-13, 2001
• 10: Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.161, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166986]
• 11: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 12: Jiming Liu , Ning Zhong, Introduction: a journal pivotal to the future of web intelligence and agent systems, Web Intelligence and Agent System, v.1 n.1, p.1-1, January 2003
• 13: McNeill, D. Hand and Mind. The University of Chicago Press, 1992.
• 14: McNeill, D., Quek, F., McCullough, K.-E., Duncan, S., Furuyama, N., Bryll, R., Ma, X.-F., and Ansari, R. Catchments, prosody, and discourse. Gesture 1 (2001), 9--33.
• 15: Nakano, Y. I., Okamoto, M., Kawahara, D., Li, Q., and Nishida, T. Converting a text into agent animations: Assigning gestures to a text. HLT-NAACL 2004: Companion Volume. ACL Press (2004), 153--156.
• 16: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 17: Rennie, J. D. M., Shih, L., Teevan, J., and Karger, D. R. Tackling the poor assumptions of naive bayes text classifiers. Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003). AAAI Press (2003).
• 18: Sparrell, C. Coverbal iconic gesture in human-computer interaction. Master's thesis, Massachusetts Institute of Technology, 1993.
• 19: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 20: Yingen Xiong , Francis Quek , David McNeill, Hand motion gestural oscillations and multimodal discourse, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958459]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language; Voice I/O; Evaluation/methodology; Theory and methods

General Terms:
Experimentation, Human Factors, Reliability

Keywords:
gesture recognition, gesture taxonomies, multimodal disambiguation, validity