Multimodal expressive embodied conversational agents

ABSTRACT

In this paper we present our work toward the creation of a multimodal expressive Embodied Conversational Agent (ECA). Our agent, called Greta, exhibits nonverbal behaviors synchronized with speech. We are using the taxonomy of communicative functions developed by Isabella Poggi [22] to specify the behavior of the agent. Based on this taxonomy a representation language, Affective Presentation Markup Language, APML has been defined to drive the animation of the agent [4]. Lately, we have been working on creating no longer a generic agent but an agent with individual characteristics. We have been concentrated on the behavior specification for an individual agent. In particular we have defined a set of parameters to change the expressivity of the agent's behaviors. Six parameters have been defined and implemented to encode gesture and face expressivity. We have performed perceptual studies of our expressivity model.

REFERENCES

• 1: Elisabetta Bevacqua , Catherine Pelachaud, Expressive audio-visual speech: Research Articles, Computer Animation and Virtual Worlds, v.15 n.3-4, p.297-304, July 2004 [doi>10.1002/cav.v15:3/4]
• 2: A. Black, P. Taylor, R. Caley, and R. Clark. Festival. http://www.cstr.ed.ac.uk/projects/festival/.
• 3: A. J. Blanchard and L. Canamero. Using visual velocity detection to achieve sychronization in imitation. In Workshop Imitation in AISB'05, 2005.
• 4: B. DeCarolis, C. Pelachaud, I. Poggi, and M. Steedman. APML, a mark-up language for believable behavior generation. In H. Prendinger and M. Ishizuka, editors, Life-like Characters. Tools, Affective Functions and Applications, pages 65--85. Springer, 2004.
• 5: P. Ekman. About brows: Emotional and conversational signals. In M. von Cranach, K. Foppa, W. Lepenies, and D. Ploog, editors, Human ethology: Claims and limits of a new discipline: contributions to the Colloquium, pages 169--248. Cambridge University Press, Cambridge, England; New-York, 1979.
• 6: Irfan Essa , Sumit Basu , Trevor Darrell , Alex Pentland, Modeling, Tracking and Interactive Animation of Faces and Heads Using Input from Video, Proceedings of the Computer Animation, p.68, June 03-04, 1996
• 7: P. E. Gallaher. Individual differences in nonverbal behavior: Dimensions of style. Journal of Personality and Social Psychology, 63(1):133--145, 1992.
• 8: Jonathan Gratch , Stacy Marsella, Tears and fears: modeling emotions and emotional behaviors in synthetic agents, Proceedings of the fifth international conference on Autonomous agents, p.278-285, May 2001, Montreal, Quebec, Canada [doi>10.1145/375735.376309]
• 9: B. Hartmann , M. Mancini , S. Buisine , C. Pelachaud, Design and evaluation of expressive gesture synthesis for embodied conversational agents, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082640]
• 10: Björn Hartmann , Maurizio Mancini , Catherine Pelachaud, Formational Parameters and Adaptive Prototype Instantiation for MPEG-4 Compliant Gesture Synthesis, Proceedings of the Computer Animation, p.111, June 19-21, 2002
• 11: B. Hartmann, M. Mancini, and C. Pelachaud. Implementing expressive gesture synthesis for embodied conversational agents. In Gesture Workshop, Vannes, May 2005.
• 12: Kristina Höök , Adrian Bullock , Ana Paiva , Marco Vala , Ricardo Chaves , Rui Prada, FantasyA and SenToy, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766002]
• 13: S. Marsella, W. Johnson, and C. LaBore. Interactive pedagogical drama for health interventions. In 11th International Conference on Artificial Intelligence in Education, Sydney, Australia, July 2003.
• 14: D. McNeill. Hand and Mind: What Gestures Reveal about Thought. University of Chicago, 1992.
• 15: A. Paiva, J. Dias, D. Sobral, S. Woods, and L. Hall. Building empathic life-like characters: the proximity factor. In Workshop on Empathic Agents, AAMAS'04, 2004.
• 16: C. Pelachaud. Visual text-to-speech. In I. S. Pandzic and R. Forcheimer, editors, MPEG4 Facial Animation - The standard, implementations and applications. John Wiley & Sons, 2002.
• 17: C. Pelachaud and M. Bilvi. Modelling gaze behavior for conversational agents. In proceedings of the IVA 2003 conference. Springer LINAI Series, 2003.
• 18: Catherine Pelachaud , Valeria Carofiglio , Berardina De Carolis , Fiorella de Rosis , Isabella Poggi, Embodied contextual agent in information delivering application, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2, July 15-19, 2002, Bologna, Italy [doi>10.1145/544862.544921]
• 19: C. Pelachaud and I. Poggi. Subtleties of facial expressions in embodied agents. Journal of Visualization and Computer Animation, 13:301--312, 2002.
• 20: C. Peters. Direction of attention perception for conversation initiation in virtual environments. In The 5th International Working Conference on Intelligent Virtual Agents, Kos, Greece, September 12-14 2005.
• 21: C. Peters, C. Pelachaud, E. Bevacqua, I. Poggi, and M. Mancini. A model of attention and interest using gaze behavior. In The 5th International Working Conference on Intelligent Virtual Agents, Kos, Greece, September 12-14 2005.
• 22: I. Poggi. Mind markers. In N. T. M. Rector, I. Poggi, editor, Gestures. Meaning and use. University Fernando Pessoa Press, Oporto, Portugal, 2003.
• 23: Isabella Poggi , Catherine Pelachaud, Emotional meaning and expression in animated faces, Affective interactions: towards a new generation of computer interfaces, Springer-Verlag New York, Inc., New York, NY, 2001
• 24: Isabella Poggi , Catherine Pelachaud , Fiorella de Rosis, Eye communication in a conversational 3D synthetic agent, AI Communications, v.13 n.3, p.169-181, January 1, 2001
• 25: Byron Reeves , Clifford Nass, The media equation: how people treat computers, television, and new media like real people and places, Cambridge University Press, New York, NY, 1996
• 26: Candace L. Sidner , Cory D. Kidd , Christopher Lee , Neal Lesh, Where to look: a study of human-robot engagement, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964458]
• 27: M. Steedman. Using APML to specify intonation. Unpublished Tutorial Paper, 2004.
• 28: H. G. Wallbott and K. R. Scherer. Cues and channels in emotion recognition. Journal of Personality and Social Psychology, 51(4):690--699, 1986.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Algorithms

Keywords:
expressive gesture, interactive embodied conversational agent, multimodal interface