SmartBody: behavior realization for embodied conversational agents

ABSTRACT

Researchers demand much from their embodied conversational agents (ECAs), requiring them to be both life-like, as well as responsive to events in an interactive setting. We find that a flexible combination of animation approaches may be needed to satisfy these needs. In this paper we present SmartBody, an open source modular framework for animating ECAs in real time, based on the notion of hierarchically connected animation controllers. Controllers in SmartBody can employ arbitrary animation algorithms such as keyframe interpolation, motion capture or procedural animation. Controllers can also schedule or combine other controllers. We discuss our architecture in detail, including how we incorporate traditional approaches, and develop the notion of a controller as a reactive module within a generic framework, for realizing modular animation control. To illustrate the versatility of the architecture, we also discuss a range of applications that have used SmartBody successfully.

REFERENCES

• 1: Marcelo Kallmann , Stacy Marsella, Hierarchical motion controllers for real-time autonomous virtual humans, Lecture Notes in Computer Science, Springer-Verlag, London, 2005 [doi>10.1007/11550617_22]
• 2: Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., Thorisson, K., Vilhjalmsson, H.: Towards a common framework for multimodal generation: The behavior markup language. In: Proc. of Intelligent Virtual Agents (IVA'06). (2006) 105--111
• 3: Lee, J., Marsella, S.: Nonverbal behavior generator for embodied conversational agents. In: Proc. of Intelligent Virtual Agents (IVA'06), Marina del Rey, CA (August 21-23 2006) 243--255
• 4: P. Ekman, W. V. F.: InvestigatorÕs guide to the Facial Action Coding System. Consulting Psychologist Press (1978)
• 5: Ekman, P.: Emotion in the human face. Cambridge University Press (1982)
• 6: Remi Arnaud , Mark C. Barnes, Collada: Sailing the Gulf of 3d Digital Content Creation, AK Peters Ltd, 2006
• 7: William Swartout , Jonathan Gratch , Randall W. Hill , Eduard Hovy , Stacy Marsella , Jeff Rickel , David Traum, Toward virtual humans, AI Magazine, v.27 n.2, p.96-108, July 2006
• 8: Hill, R., Belanich, Core, M., Lane, Dixon, Forbell, E., Kim, J., Hart: Pedagogically structured game-based training: Development of the elect bilat simulation. In: Poster presentation at the 25th Army Science Conference, Orlando, FL. (Nov 2006)
• 9: Kenny, P., Parsons, T. D., Gratch, J., Rizzo, A.: Virtual patients for clinical therapist skills training (2007)
• 10: Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., van der Werf, R. J., Morency, L. P.: Virtual rapport. In: Proc. of Intelligent Virtual Agents (IVA'06), Marina del Rey, CA (August 21-23 2006)
• 11: Ronan Boulic , Nadia Magnenat-Thalmann , Daniel Thalmann, A global human walking model with real-time kinematic personification, The Visual Computer: International Journal of Computer Graphics, v.6 n.6, p.344-358, Dec. 1990 [doi>10.1007/BF01901021]
• 12: Sang Il Park , Hyun Joon Shin , Sung Yong Shin, On-line locomotion generation based on motion blending, Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 21-22, 2002, San Antonio, Texas [doi>10.1145/545261.545279]
• 13: Kallmann, M.: Scalable solutions for interactive virtual humans that can manipulate objects. In: Artificial Intelligence and Interactive Digital Entertainment (AIIDE), Marina del Rey, CA (June 1-3 2005)
• 14: Ying Liu , Norman I. Badler, Real-Time Reach Planning for Animated Characters Using Hardware Acceleration, Proceedings of the 16th International Conference on Computer Animation and Social Agents (CASA 2003), p.86, May 08-09, 2003
• 15: James Kuffner, Jr. , Jean-Claude Latombe, Interactive Manipulation Planning for Animated Characters, Proceedings of the 8th Pacific Conference on Computer Graphics and Applications, p.417, October 03-05, 2000
• 16: Baerlochcer, P.: Inverse Kinematics Techniques for the Interactive Posture Control of Articulated Figures. PhD thesis, Swiss Federal Institute of Technology, EPFL (2001) Thesis number 2383.
• 17: Tolani, D., Badler, N.: Real-time inverse kinematics of the human arm. Presence 5(4) (1996) 393--401
• 18: Gu, E., Badler, N.: Visual attention and eye gaze during multipartite conversations with distractions. In: Proc. of Intelligent Virtual Agents (IVA'06), Marina del Rey, CA (2006)
• 19: Bodenheimer, B., Rose, C., Rosenthal, S., Pella, J.: The process of motion capture: Dealing with the data. In Thalmann, D., van de Panne, M., eds.: Computer Animation and Simulation '97, Springer NY (Sept 1997) 3--18 Eurographics Animation Workshop.
• 20: Thoroughman, K. A., Shadmehr, R.: Learning of action through combination of motor primitives. Nature 407(6805) (2000) 742--747
• 21: Giszter, S. F., Mussa-Ivaldi, F. A., Bizzi, E.: Convergent force fields organized in the frogŠs spinal cord. Journal of Neuroscience 13(2) (1993) 467--491
• 22: John P. Granieri , Jonathan Crabtree , Norman I. Badler, Production and playback of human figure motion for visual simulation, ACM Transactions on Modeling and Computer Simulation (TOMACS), v.5 n.3, p.222-241, July 1995 [doi>10.1145/217853.217856]
• 23: Ken Perlin , Athomas Goldberg, Improv: a system for scripting interactive actors in virtual worlds, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.205-216, August 1996 [doi>10.1145/237170.237258]
• 24: Ronan Boulic , Pascal Bécheiraz , Luc Emering , Daniel Thalmann, Integration of motion control techniques for virtual human and avatar real-time animation, Proceedings of the ACM symposium on Virtual reality software and technology, p.111-118, September 1997, Lausanne, Switzerland [doi>10.1145/261135.261156]
• 25: Petros Faloutsos , Michiel van de Panne , Demetri Terzopoulos, Composable controllers for physics-based character animation, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.251-260, August 2001 [doi>10.1145/383259.383287]
• 26: Stefan Kopp , Ipke Wachsmuth, Synthesizing multimodal utterances for conversational agents: Research Articles, Computer Animation and Virtual Worlds, v.15 n.1, p.39-52, March 2004 [doi>10.1002/cav.v15:1]
• 27: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 28: Carolis, B. D., Pelachaud, C., Poggi, I., de Rosis, F.: Behavior planning for a reflexive agent. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'01), Seattle (Sept 2001)
• 29: Matthew Stone , Doug DeCarlo , Insuk Oh , Christian Rodriguez , Adrian Stere , Alyssa Lees , Chris Bregler, Speaking with hands: creating animated conversational characters from recordings of human performance, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Intelligent agents

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality; Animation

General Terms:
Algorithms, Design, Theory

Keywords:
character animation, conversational characters, virtual humans