Towards integrated microplanning of language and iconic gesture for multimodal output

ABSTRACT

When talking about spatial domains, humans frequently accompany their explanations with iconic gestures to depict what they are referring to. For example, when giving directions, it is common to see people making gestures that indicate the shape of buildings, or outline a route to be taken by the listener, and these gestures are essential to the understanding of the directions. Based on results from an ongoing study on language and gesture in direction-giving, we propose a framework to analyze such gestural images into semantic units (image description features), and to link these units to morphological features (hand shape, trajectory, etc.). This feature-based framework allows us to generate novel iconic gestures for embodied conversational agents, without drawing on a lexicon of canned gestures. We present an integrated microplanner that derives the form of both coordinated natural language and iconic gesture directly from given communicative goals, and serves as input to the speech and gesture realization engine in our NUMACK project.

REFERENCES

• 1: Cassell, J. & Prevost, S. Distribution of Semantic Features Across Speech and Gesture by Humans and Computers. In Proc. Workshop on Integration of Gesture in Language and Speech, 1996, Wilmington, DE.
• 2: Cassell, J., McNeill, D. & McCullough, K.E. Speech-Gesture Mismatches: Evidence for One Underlying Representation of Linguistic and Non-Linguistic Information. Pragmatics and Cognition 7(1): 1--33, 1999.
• 3: Justine Cassell , Matthew Stone , Hao Yan, Coordination and context-dependence in the generation of embodied conversation, Proceedings of the first international conference on Natural language generation, June 12-16, 2000, Mitzpe Ramon, Israel [doi>10.3115/1118253.1118277]
• 4: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 5: Diane Chi , Monica Costa , Liwei Zhao , Norman Badler, The EMOTE model for effort and shape, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.173-182, July 2000 [doi>10.1145/344779.352172]
• 6: Clark, H. H. Using Language. Cambridge Univ. Press, 1996.
• 7: de Ruiter, J.P. The production of gesture and speech. In McNeill, D. (ed.) Language and Gesture. Cambridge, UK: Cambridge University Press, 2000.
• 8: Karen Emmorey , Barbara Tversky , Holly A. Taylor, Using space to describe space: Perspective inspeech, sign, and gesture, Spatial Cognition and Computation, v.2 n.3, p.157-180, 2000 [doi>10.1023/A:1013118114571]
• 9: Gao, Y. Automatic extraction of spatial location for gesture generation, Master thesis, MIT Dept. of Electrical Engineering and Computer Science, 2002.
• 10: Green, N., G. Carenini, et al. A Media-Independent Content Language for Integrated Text and Graphics Generation. Workshop on Content Visualization and Intermedia Representations at COLING/ACL '98, Montreal, 1998.
• 11: Annette Herskovits, Language and Spatial Cognition, Cambridge University Press, New York, NY, 1987
• 12: Joshi, A.K. An Introduction to Tree Adjoining Grammars, In A. Manaster-Ramer (ed), Mathematics of Language, Amsterdam: John Benjamins, pp. 87--114, 1987.
• 13: Stephan Kerpedjiev , Giuseppe Carenini , Steven F. Roth , Johanna D. Moore, AutoBrief: a multimedia presentation system for assisting data analysis, Computer Standards & Interfaces, v.18 n.6-7, p.583-593, Dec. 1997 [doi>10.1016/S0920-5489(97)00022-6]
• 14: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 15: Stefan Kopp , Ipke Wachsmuth, Synthesizing multimodal utterances for conversational agents: Research Articles, Computer Animation and Virtual Worlds, v.15 n.1, p.39-52, March 2004 [doi>10.1002/cav.v15:1]
• 16: Landau, B. & Jackendoff, R. What and where in spatial language and spatial cognition. Behavioral and Brain Sciences 16: 217--265, 1993.
• 17: McNeill, D. & Levy, E. Conceptual representations in language activity and gesture. In R. Jarvella, & W. Klein (eds.): Speech, Place, and Action, John Wiley & Sons, 1982.
• 18: McNeill, D., Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
• 19: McNeill, D. Catchments and Contexts: Non-modular factors in speech and gesture production. In D. McNeill (ed.): Language and Gesture. Cambridge University Press, 2000.
• 20: Nijholt, A., Theune , M. & Heylen, D. Embodied Language Generation, In O. Stock & M. Zancanaro (eds): Intelligent Information Presentation, Kluwer, 2004.
• 21: Ken Perlin , Athomas Goldberg, Improv: a system for scripting interactive actors in virtual worlds, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.205-216, August 1996 [doi>10.1145/237170.237258]
• 22: Pelachaud, C. & Poggi, I. Multimodal Embodied Agents, In Autonomous Agents Workshop Multimodal Communication and Context in Embodied Agents, pp. 95--99, 2001.
• 23: Ehud Reiter , Robert Dale, Building natural language generation systems, Cambridge University Press, New York, NY, 2000
• 24: Jeff Rickel , Stacy Marcella , Jonathan Gratch , Randall Hill , David Traum , William Swartout, Toward a New Generation of Virtual Humans for Interactive Experiences, IEEE Intelligent Systems, v.17 n.4, p.32-38, July 2002 [doi>10.1109/MIS.2002.1024750]
• 25: Sowa, T. & Wachsmuth, I. Coverbal Iconic Gestures for Object Descriptions in Virtual Environments: An Empirical Study. In M. Rector, I. Poggi & N. Trigo (eds.): Proc. "Gestures. Meaning and Use", pp. 365--376, 2003.
• 26: Stone, M., Doran, C., Webber, B., Bleam, T. & Palmer, M. Microplanning with communicative intentions: the SPUD system. Computational Intelligence 19(4): 311--381, 2003.
• 27: Stuart G. Towns , Charles B. Callaway , James C. Lester, Generating coordinated natural language and 3D animations for complex spatial explanations, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.112-119, July 1998, Madison, Wisconsin, United States
• 28: David Traum , Jeff Rickel, Embodied agents for multi-party dialogue in immersive virtual worlds, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2, July 15-19, 2002, Bologna, Italy [doi>10.1145/544862.544922]
• 29: Yan, H. Paired Speech and Gesture Generation in Embodied Conversational Agents. Masters Thesis. MIT, School of Architecture and Planning, 2000.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language generation

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

General Terms:
Algorithms, Experimentation, Languages, Theory

Keywords:
embodied conversational agents, generation, gesture, language, multimodal output