BEAT: the Behavior Expression Animation Toolkit

ABSTRACT

The Behavior Expression Animation Toolkit (BEAT) allows animators to input typed text that they wish to be spoken by an animated human figure, and to obtain as output appropriate and synchronized nonverbal behaviors and synthesized speech in a form that can be sent to a number of different animation systems. The nonverbal behaviors are assigned on the basis of actual linguistic and contextual analysis of the typed text, relying on rules derived from extensive research into human conversational behavior. The toolkit is extensible, so that new rules can be quickly added. It is designed to plug into larger systems that may also assign personality profiles, motion characteristics, scene constraints, or the animation styles of particular animators.

REFERENCES

• 1: Kenji Amaya , Armin Bruderlin , Tom Calvert, Emotion from motion, Proceedings of the conference on Graphics interface '96, p.222-229, May 1996, Toronto, Ontario, Canada
• 2: Norman I. Badler , Rama Bindiganavale , Jan Allbeck , William Schuler , Liwei Zhao , Martha Palmer, Parameterized action representation for virtual human agents, Embodied conversational agents, MIT Press, Cambridge, MA, 2001
• 3: Becheiraz, P. and Thalmann, D., A Behavioral Animation System for Autonomous Actors personified by Emotions, Proc. of the1st Workshop on Embodied Conversational Characters, 57-65, 1998.
• 4: Bruce M. Blumberg , Tinsley A. Galyean, Multi-level direction of autonomous creatures for real-time virtual environments, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.47-54, September 1995 [doi>10.1145/218380.218405]
• 5: Charles Rose , Michael F. Cohen , Bobby Bodenheimer, Verbs and Adverbs: Multidimensional Motion Interpolation, IEEE Computer Graphics and Applications, v.18 n.5, p.32-40, September 1998 [doi>10.1109/38.708559]
• 6: Matthew Brand, Voice puppetry, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.21-28, July 1999 [doi>10.1145/311535.311537]
• 7: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 8: Tom Calvert, Composition of realistic animation sequences for multiple human figures, Making them move: mechanics, control, and animation of articulated figures, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1991
• 9: Justine Cassell, Nudge nudge wink wink: elements of face-to-face conversation for embodied conversational agents, Embodied conversational agents, MIT Press, Cambridge, MA, 2001
• 10: Justine Cassell , Catherine Pelachaud , Norman Badler , Mark Steedman , Brett Achorn , Tripp Becket , Brett Douville , Scott Prevost , Matthew Stone, Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.413-420, July 1994 [doi>10.1145/192161.192272]
• 11: Cassell, J. and Prevost, S., Distribution of Semantic Features Across Speech and Gesture by Humans and Computers. Proc. Workshop on the Integration of Gesture in Language and Speech, pp. 253-270, Newark, DE, 1996.
• 12: Cassell, J., Torres, O., and Prevost, S., Turn Taking vs. Discourse Structure: How Best to Model Multimodal Conversation, in Machine Conversations, Y. Wilks, Ed. The Hague: Kluwer, pp. 143-154, 1999.
• 13: Chang, J., Action Scheduling in Humanoid Conversational Agents, M.S. Thesis in Electrical Engineering and Computer Science. Cambridge, MA: MIT, 1998.
• 14: Diane Chi , Monica Costa , Liwei Zhao , Norman Badler, The EMOTE model for effort and shape, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.173-182, July 2000 [doi>10.1145/344779.352172]
• 15: Colburn, A., Cohen, M. F., and Drucker, S., The Role of Eye Gaze in Avatar Mediated Conversational Interfaces, MSR-TR-2000-81. Microsoft Research, 2000.
• 16: Halliday, M. A. K., Explorations in the Functions of Language. London: Edward Arnold, 1973.
• 17: Hirschberg, J., Accent and Discourse Context: Assigning Pitch Accent in Synthetic Speech. Proc. AAAI 90, pp. 952-957, 1990.
• 18: Hiyakumoto, L., Prevost, S., and Cassell, J., Semantic and Discourse Information for Text-to-Speech Intonation. Proc. ACL Workshop on Concept-to-Speech Generation, Madrid, 1997.
• 19: Huang, X., Acero, A., Adcock, J., Hon, H.-W., Goldsmith, J., Liu, J., and Plumpe, M., Whistler: A Trainable Text-to-Speech System. Proc. 4th Int'l. Conf. on Spoken Language Processing (ICSLP '96), pp. 2387- 2390, Piscataway, NJ, 1996.
• 20: David Kurlander , Tim Skelly , David Salesin, Comic Chat, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.225-236, August 1996 [doi>10.1145/237170.237260]
• 21: Douglas B. Lenat , R. V. Guha, Building Large Knowledge-Based Systems; Representation and Inference in the Cyc Project, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 22: Massaro, D. W., Perceiving Talking Faces: From Speech Perception to a Behavioral Principle. Cambridge, MA: MIT Press, 1987.
• 23: McNeill, D., Hand and Mind: What Gestures Reveal about Thought. Chicago, IL/London, UK: The University of Chicago Press, 1992.
• 24: Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K., Introduction to Wordnet: An On-line Lexical Database, 1993.
• 25: Katashi Nagao , Akikazu Takeuchi, Speech dialogue with facial displays: multimodal human-computer conversation, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.102-109, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981747]
• 26: A Pearce , B Wyvill , G Wyvill , D Hill, Speech and expression: a computer solution to face animation, Proceedings on Graphics Interface '86/Vision Interface '86, p.136-140, August 1986, Vancouver, British Columbia, Canada
• 27: Pelachaud, C., Badler, N., and Steedman, M., Generating Facial Expressions for Speech, Cognitive Science, 20(1), pp. 1-46, 1994.
• 28: Perlin, K., Noise, Hypertexture, Antialiasing and Gesture, in Texturing and Modeling, A Procedural Approach, D. Ebert, Ed. Cambridge, MA: AP Professional, 1994.
• 29: Ken Perlin , Athomas Goldberg, Improv: a system for scripting interactive actors in virtual worlds, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.205-216, August 1996 [doi>10.1145/237170.237258]
• 30: Scott Prevost , Mark Steedman, Specifying intonation from context for speech synthesis, Speech Communication, v.15 n.1-2, p.139-153, Oct. 1994 [doi>10.1016/0167-6393(94)90048-5]
• 31: Roehl, B., Specification for a Standard Humanoid, Version 1.1, H. A. W. Group, Ed. http://ece.uwaterloo.ca/~h-anim/spec1.1/, 1999.
• 32: Taylor, P., Black, A., and Caley, R., The architecture of the Festival Speech Synthesis System. Proc. 3rd ESCA Workshop on Speech Synthesis, pp. 147-151, Jenolan Caves, Australia, 1998.
• 33: K. Waters , T. Levergood, An automatic lip-synchronization algorithm for synthetic faces, Proceedings of the second ACM international conference on Multimedia, p.149-156, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192644]
• 34: Yan, H., Paired Speech and Gesture Generation in Embodied Conversational Agents, M.S. thesis in the Media Lab. Cambridge, MA: MIT, 2000.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Animation

General Terms:
Design, Theory

Keywords:
animation systems, facial animation, gesture, speech synthesis