A model of gaze for the purpose of emotional expression in virtual embodied agents

ABSTRACT

Currently, state of the art virtual agents lack the ability to display emotion as seen in actual humans, or even in hand-animated characters. One reason for the emotional inexpressiveness of virtual agents is the lack of emotionally expressive gaze manner. For virtual agents to express emotion that observers can empathize with, they need to generate gaze - including eye, head, and torso movement - to arbitrary targets, while displaying arbitrary emotional states. Our previous work [18] describes the Gaze Warping Transformation, a method of generating emotionally expressive head and torso movement during gaze shifts that is derived from human movement data. Through an evaluation, it was shown that applying different transformations to the same gaze shift could modify the affective state perceived when the transformed gaze shift was viewed by a human observer. In this paper we propose a model of realistic, emotionally expressive gaze that builds upon the Gaze Warping Transformation by improving the transformation implementation, and by adding a model of eye movement drawn from the visual neuroscience literature. We describe how to generate a gaze to an arbitrary target, while displaying an arbitrary emotional behavior. Finally, we propose an evaluation to determine what emotions human observers will attribute to the generated gaze shifts. Once this work is completed, virtual agents will have access to a new channel for emotionally expressive behavior.

REFERENCES

• 1: Kenji Amaya , Armin Bruderlin , Tom Calvert, Emotion from motion, Proceedings of the conference on Graphics interface '96, p.222-229, May 1996, Toronto, Ontario, Canada
• 2: Argyle, M., Cook, M. 1976. Gaze and Mutual Gaze. Cambridge University Press, (1976).
• 3: Bickmore, T., Cassell, J. 2004. Social Dialogue with Embodied Conversational Agents. Bernsen, N. (ed.) Natural, Intelligent and Effective Interaction with Multimodal Dialogue Systems. Kluwer Academic (2004). 23--54.
• 4: Matthew Brand , Aaron Hertzmann, Style machines, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.183-192, July 2000 [doi>10.1145/344779.344865]
• 5: Carlos Busso , Zhigang Deng , Ulrich Neumann , Shrikanth Narayanan, Natural head motion synthesis driven by acoustic prosodic features: Virtual Humans and Social Agents, Computer Animation and Virtual Worlds, v.16 n.3-4, p.283-290, July 2005 [doi>10.1002/cav.v16:3/4]
• 6: Carney, D. 2005. Beliefs About the Nonverbal Expression of Social Power. Journal of Nonverbal Behavior. Vol 29, No. 2 (2005). 105--123.
• 7: Coulson, M. 2004. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. Journal of Nonverbal Behavior Vol. 28, No. 2.
• 8: Ellen Douglas-Cowie , Nick Campbell , Roddy Cowie , Peter Roach, Emotional speech: towards a new generation of databases, Speech Communication, v.40 n.1-2, p.33-60, April 2003 [doi>10.1016/S0167-6393(02)00070-5]
• 9: Exline, R. 1974. Visual Interaction: The Glances of Power and Preference. Weitz, S. (ed.) Nonverbal Communication: Readings with Commentary, Oxford University Press, Oxford (1974).
• 10: Atsushi Fukayama , Takehiko Ohno , Naoki Mukawa , Minako Sawaki , Norihiro Hagita, Messages embedded in gaze of interface agents --- impression management with agent's gaze, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503385]
• 11: Patrick Gebhard, ALMA: a layered model of affect, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082478]
• 12: Gresty, M. A. 1974. Coordination of Head and Eye Movements to Fixate Continuous and Intermittent Targets. Vision Research, Vol 14. (1974). 395--403.
• 13: Guitton, D., Volle, M. 1987. Gaze Control in Humans: Eye-Head Coordination During Orienting Movements to Targets Within and Beyond the Oculomotor Range. Journal of Neurophysiology. Vol 58, No. 3. (Sep. 1987). 427--459.
• 14: Itti, L., Dhavale, N., Pighin, F. 2003. Realistic Avatar Eye and Head Animation using a Neurobiological Model of Visual Attention. SPIE 48^th Annual International Symposium on Optical Science and Technology. 2003.
• 15: Michael Kipp , Michael Neff , Kerstin H. Kipp , Irene Albrecht, Towards Natural Gesture Synthesis: Evaluating Gesture Units in a Data-Driven Approach to Gesture Synthesis, Proceedings of the 7th international conference on Intelligent Virtual Agents, September 17-19, 2007, Paris, France [doi>10.1007/978-3-540-74997-4_2]
• 16: Kleinke, C. 1986. Gaze and Eye Contact: A Research Review. Psychological Bulletin. Vol. 100, No. 1. (1986) 78--100.
• 17: Lance, B., Marsella, S., Koizumi, D. 2004. Towards Expressive Gaze Manner in Embodied Virtual Agents (2004)
• 18: Brent Lance , Stacy C. Marsella, Emotionally Expressive Head and Body Movement During Gaze Shifts, Proceedings of the 7th international conference on Intelligent Virtual Agents, September 17-19, 2007, Paris, France [doi>10.1007/978-3-540-74997-4_8]
• 19: Sooha Park Lee , Jeremy B. Badler , Norman I. Badler, Eyes alive, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 20: Leigh, R. J., Zee, D. 2006. The Neurology of Eye Movements, 4^th Ed. Oxford University Press, (2006).
• 21: C. Karen Liu , Aaron Hertzmann , Zoran Popović, Learning physics-based motion style with nonlinear inverse optimization, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 22: Mehrabian, A. 1981. Silent Messages: Implicit Communication of Emotions and Attitudes, 2^nd edn. Wadsworth Publishing Company (1981).
• 23: Mehrabian, A., Ksionzky, S. 1972. Some Determiners of Social Interaction. Sociometry, Vol 35. (1972). 588--609.
• 24: de Meijer, M. 1989. The Contribution of General Features of Body Movement to the Attribution of Emotions. Journal of Nonverbal Behavior. Vol 13, No. 4 (1989). 247--268.
• 25: Mignault, A., Chaudhuri, A. 2003. The Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion. Journal of Nonverbal Behavior Vol. 27 No.2. (Summer 2003).
• 26: Paterson, H., Pollick, F., Sanford, A. 2001. The Role of Velocity in Affect Discrimination. Proceedings of the 23rd Annual Conference of the Cognitive Science Society (2001).
• 27: Pelachaud, C., Bilvi, M. 2003. Modelling Gaze Behavior for Conversational Agents. Proceedings of Intelligent Virtual Agents 2003. LINAI Series, Springer, Heidelberg (2003).
• 28: Pelachaud, C., Poggi, I. 2002. Subtleties of facial expressions in embodied agents. The Journal of Visualization and Computer Animation. Vol 13, No. 5. (2002). 301--312.
• 29: Rickel, J., Johnson, W. L. 1999. Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control. Applied Artificial Intelligence 13(4-5). (1999). 343--382.
• 30: Sobczynski, P. 2004. "Polar Express, The". http://www.efilmcritic.com/review.php?movie=10918&reviewer=389. November 2004.
• 31: Stark, L., Zangemeister, W. H., Edwards, J., Grinberg, J., Jones, A., Lehman, S., Lubock, P., Narayan, V., Nystrom, M. 1980. Head Rotation Trajectories Compared with Eye Saccades by Main Sequence Relationships. Investigative Ophthalmology and Visual Science. Vol 19, No. 8. (1980). 986--988.
• 32: Thomas, F., Johnston, O. The Illusion of Life: Disney Animation. Walt Disney Productions. (1981).
• 33: Wallbott, H. G. 1998. Bodily Expression of Emotion. European Journal of Social Psychology. Vol 28 (1998). 879--896.
• 34: Andrew Witkin , Zoran Popovic, Motion warping, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.105-108, September 1995 [doi>10.1145/218380.218422]
• 35: Zee, D. 1976. Disorders of Eye-Head Coordination. Brooks, B. A., Bajandas, F. J. Eye Movements: ARVO Symposium. Plenium Press. (1976). 9--40.
• 36: Jianmin Zhao , Norman I. Badler, Inverse kinematics positioning using nonlinear programming for highly articulated figures, ACM Transactions on Graphics (TOG), v.13 n.4, p.313-336, Oct. 1994 [doi>10.1145/195826.195827]
• 37: Liwei Zhao , Norman I. Badler, Acquiring and validating motion qualities from live limb gestures, Graphical Models, v.67 n.1, p.1-16, January 2005 [doi>10.1016/j.gmod.2004.08.002]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

General Terms:
Human Factors

Keywords:
animation, emotional expression, gaze, motion capture, nonverbal, posture, virtual agent