Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks

ABSTRACT

Embodied conversational agents are required to be able to express themselves convincingly and autonomously. Based on an empirial study on spatial descriptions of landmarks in direction-giving, we present a model that allows virtual agents to automatically generate, i.e., select the content and derive the form of coordinated language and iconic gestures. Our model simulates the interplay between these two modes of expressiveness on two levels. First, two kinds of knowledge representation (propositional and imagistic) are utilized to capture the modality-specific contents and processes of content planning. Second, specific planners are integrated to carry out the formulation of concrete verbal and gestural behavior. A probabilistic approach to gesture formulation is presented that incorporates multiple contextual factors as well as idiosyncratic patterns in the mapping of visuo-spatial referent properties onto gesture morphology. Results from a prototype implementation are described.

REFERENCES

• 1: J. Bavelas, J. Gerwing, C. Sutton, and D. Prevost. Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58:495--520, 2008.
• 2: Justine Cassell , Matthew Stone , Hao Yan, Coordination and context-dependence in the generation of embodied conversation, Proceedings of the first international conference on Natural language generation, June 12-16, 2000, Mitzpe Ramon, Israel [doi>10.3115/1118253.1118277]
• 3: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 4: J. de Ruiter. The production of gesture and speech. In D. McNeill, editor, Language and gesture. Cambridge University Press, 2000.
• 5: M. Denis. The description of routes: A cognitive approach to the production of spatial discourse. Current Psychology of Cognition, 16:409--458, 1997.
• 6: B. Hartmann, M. Mancini, and C. Pelachaud. Implementing expressive gesture synthesis for embodied conversational agents. In Gesture in Human-Computer Interaction and Simulation, 2005.
• 7: A. Hostetter, M. Alibali, and S. Kita. Does sitting on your hands make you bite your tongue? The effects of gesture inhibition on speech during motor descriptions. In D. S. McNamara and J. G. Trafton, editors, Proc. 29th meeting of the Cognitive Science Society, pages 1097--1102. Erlbaum, 2007.
• 8: Matt Huenerfauth , Liming Zhao , Erdan Gu , Jan Allbeck, Design and evaluation of an American Sign Language generator, Proceedings of the Workshop on Embodied Language Processing, p.51-58, June 29-29, 2007, Prague, Czech Republic
• 9: A. Kendon. Gesture--Visible Action as Utterance. Cambridge University Press, 2004.
• 10: S. Kita and A. Özyürek. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48:16--32, 2003.
• 11: S. Kopp, K. Bergmann, and I. Wachsmuth. Multimodal communication from multimodal thinking - towards an integrated model of speech and gesture production. International Journal of Semantic Computing, 2(1):115--136, 2008.
• 12: S. Kopp, P. Tepper, K. Ferriman, K. Striegnitz, and J. Cassell. Trading spaces: How humans and humanoids use speech and gesture to give directions. In T. Nishida, editor, Conversational Informatics, chapter 8, pages 133--160. John Wiley, 2007.
• 13: Stefan Kopp , Ipke Wachsmuth, Synthesizing multimodal utterances for conversational agents: Research Articles, Computer Animation and Virtual Worlds, v.15 n.1, p.39-52, March 2004 [doi>10.1002/cav.v15:1]
• 14: S. Levinson. Frames of reference and molyneux's question: Cross-linguistic evidence. In Space and Language, pages 109--169. MIT Press, 1996.
• 15: E. Morsella and R. Krauss. The role of gestures in spatial working memory and speech. The American Journal of Psychology, 117:411--424, 2004.
• 16: Michael Neff , Michael Kipp , Irene Albrecht , Hans-Peter Seidel, Gesture modeling and animation based on a probabilistic re-creation of speaker style, ACM Transactions on Graphics (TOG), v.27 n.1, p.1-24, March 2008 [doi>10.1145/1330511.1330516]
• 17: A. Paivio. Mental Representations. Oxford Univ. Press, 1986.
• 18: T. Sowa and I. Wachsmuth. A model for the representation and processing of shape in coverbal iconic gestures. In Proc. KogWis05, pages 183--188, Basel, 2005. Schwabe.
• 19: H. Steck and V. Tresp. Bayesian belief networks for data mining. In Proceedings of the 2nd Workshop "Data Mining und Data Warehousing als Grundlage moderner enscheidungsunterstuetzender System", 1999.
• 20: Matthew Stone , Doug DeCarlo , Insuk Oh , Christian Rodriguez , Adrian Stere , Alyssa Lees , Chris Bregler, Speaking with hands: creating animated conversational characters from recordings of human performance, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 21: M. Stone, C. Doran, B. Webber, T. Bleam, and M. Palmer. Microplanning with Communicative Intentions: The Spud System. Comput. Intelligence, 19(4):311--381, 2003.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.0 General
          Subjects: Cognitive simulation

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
          Subjects: Natural language interfaces
        I.2.11 Distributed Artificial Intelligence
            Subjects: Intelligent agents

General Terms:
Design, Experimentation, Theory

Keywords:
embodied conversational agents, expressiveness, gesture, language, multimodal output