Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents

ABSTRACT

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversation is created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gestures generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout we will use examples from an actual synthesized, fully animated conversation.

REFERENCES

• 1: M. Argyle and M. Cook. Gaze and Mutual gaze. Cambridge University Press, 1976.
• 2: Norman I. Badler , Brian A. Barsky , David Zeltzer, Making them move: mechanics, control, and animation of articulated figures, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1991
• 3: Norman I. Badler , Cary B. Phillips , Bonnie Lynn Webber, Simulating humans: computer graphics animation and control, Oxford University Press, Inc., New York, NY, 1993
• 4: Welton M. Becket. The jack lisp api. Technical Report MS-CIS-94-01/Graphics Lab 59, University of Pennsylvania, 1994.
• 5: Tom Calvert, Composition of realistic animation sequences for multiple human figures, Making them move: mechanics, control, and animation of articulated figures, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1991
• 6: J. Cappella. personal communication, 1993.
• 7: Justine Cassell, Mark Steedman, Norm Badler, Catherine Pelachaud, Matthew Stone, Brett Douville, Scott Prevost and Brett Achorn. Modeling the interaction between speech and gesture. Proceedings of the Cognitive Science Society Annual Conference, 1994.
• 8: Justine Cassell and David McNeill. Gesture and the poetics of prose. Poetics Today, 12:375-404, 1992.
• 9: Justine Cassell, David McNeill, and Karl-Erik McCullough. Kids, don't try this at home: Experimental mismatches of speech and gesture. presented at the International Communication Association annual meeting, 1993.
• 10: D. T. Chen, S. D. Pieper, S. K. Singh, J. M. Rosen, and D. Zeltzer. The virtual sailor: An implementation of interactive human body modeling. In Proc. 1993 Virtual Reality Annual International Symposium, Seattle, WA, September 1993. IEEE.
• 11: M.M. Cohen and D.W. Massaro. Modeling coarticulation in synthetic visual speech. In N.M. Thalmann and D.Thalmann, editors, Models and Techniques in Computer Animation, pages 139-156. Springer-Verlag, 1993.
• 12: G. Collier. Emotional expression. Lawrence Erlbaum Associates, 1985.
• 13: W.S. Condon and W.D. Osgton.Speech and body motion synchrony of the speaker-hearer. In D.H. Hortonand J.J. Jenkins, editors, The perceptionof Language, pages 150-184. Academic Press, 1971.
• 14: S. Duncan. Some signals and rules for taking speaking turns in conversations. In Weitz, editor, Nonverbal Communication. Oxford University Press, 1974.
• 15: P. Ekman. Movements with precise meanings. The Journal of Communication, 26, 1976.
• 16: P. Ekman. About brows: emotional and conversational signals. In M. von Cranach, K. Foppa, W. Lepenies, and D. Ploog, editors, Humanethology: claims and limits of a new disipline: contributions to the Colloquium, pages 169-248. Cambridge University Press, Cambridge, England; New-York, 1979.
• 17: P. Ekmanand W. Friesen. Facial Action Coding System. Consulting Psychologists Press, Inc., 1978.
• 18: J.-P. Gourret , N. M. Thalmann , D. Thalmann, Simulation of object and human skin formations in a grasping task, ACM SIGGRAPH Computer Graphics, v.23 n.3, p.21-30, July 1989
• 19: P. Kalra, A. Mangili, N. Magnenat-Thalmann, and D. Thalmann. Smile: A multilayeredfacial animationsystem. In T.L. Kunii, editor, Modeling in Computer Graphics. Springer-Verlag, 1991.
• 20: A. Kendon. Movement coordination in social interaction: some examples de-scribed. In Weitz, editor, Nonverbal Communication. Oxford University Press, 1974.
• 21: AdamKendon. Gesticulation and speech: Two aspects of the process of utterance. In M.R.Key, editor, The Relation between Verbal and Nonverbal Communication, pages 207-227. Mouton, 1980.
• 22: Jintae Lee and Tosiyasu L. Kunii. Visual translation: From native language to sign language. In Workshop on Visual Languages, Seattle, WA, 1993. IEEE.
• 23: Philip Lee , Susanna Wei , Jianmin Zhao , Norman I. Badler, Strength guided motion, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.253-262, Aug. 1990
• 24: Mark Liberman and A. L. Buchsbaum. Structure and usage of current Bell Labs text to speech programs. Technical MemorandumTM 11225-850731-11, AT&T Bell Laboratories, 1985.
• 25: Jeffrey Loomis , Howard Poizner , Ursula Bellugi , Alynn Blakemore , John Hollerbach, Computer graphic modeling of american sign language, ACM SIGGRAPH Computer Graphics, v.17 n.3, p.105-114, July 1983
• 26: Nadia Magnenat-Thalmann , Daniel Thalmann, Human body deformations using joint-dependent local operators and finite-element theory, Making them move: mechanics, control, and animation of articulated figures, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1991
• 27: David McNeill. Handand Mind: What Gestures Reveal about Thought. University of Chicago, 1992.
• 28: M. Patel. Making FACES. PhD thesis, School of Mathematical Sciences, Univer-sity of Bath, Bath, AVON, UK, 1991.
• 29: C. Pelachaud, N.I. Badler, and M. Steedman. Linguistic issues in facial animation. In N. Magnenat-Thalmann and D. Thalmann, editors, Computer Animation '91, pages 15-30. Springer-Verlag, 1991.
• 30: Richard Power. The organisation of purposeful dialogues. Linguistics, 1977.
• 31: Scott Prevost , Mark Steedman, Generating contextually appropriate intonation, Proceedings of the sixth conference on European chapter of the Association for Computational Linguistics, April 21-23, 1993, Utrecht, The Netherlands [doi>10.3115/976744.976783]
• 32: Ellen F. Prince. The ZPG letter: Subjects, definiteness and information status. In S. Thompson and W. Mann, editors, Discourse description: diverse analyses of a fund raising text, pages 295-325. John Benjamins B.V., 1992.
• 33: Hans Rijpkema , Michael Girard, Computer animation of knowledge-based human grasping, ACM SIGGRAPH Computer Graphics, v.25 n.4, p.339-348, July 1991
• 34: Barbara Robertson. Easy motion. Computer Graphics World, 16(12):33-38, December 1993.
• 35: Klaus R. Scherer. The functions of nonverbal signs in conversation. In H. Giles R. St. Clair, editor, The Social and Physhological Contexts of Language, pages 225-243. Lawrence Erlbaum Associates, 1980.
• 36: Mark Steedman. Structure and intonation. Language, 67:260-296, 1991.
• 37: Akikazu Takeuchi , Katashi Nagao, Communicative facial displays as a new conversational modality, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.187-193, May 1993, Amsterdam, The Netherlands
• 38: K. Tuite. The production of gesture. Semiotica, 93(1/2), 1993.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Theory