Can local avatars satisfy a global audience? A case study of high-fidelity 3D facial avatar animation in subject identification and emotion perception by US and international groups

ABSTRACT

This study investigates effectiveness of a local high-fidelity 3D facial avatar for a global audience by observing how US and International student groups differed in identifying subjects and perceiving emotions while viewing nonverbal high-fidelity 3D facial avatar animations embedded with the motion data of three US individuals. To synthesize the animated 3D avatars to convey highly believable facial expressions, a 3D scanned facial model was mapped with high-fidelity motion-capture data of three native US subjects as they spoke designated English sentences with specified emotions. Simple animations in conjunction with actual footage of the subjects speaking during the facial motion-capture sessions were shown several times to both native US and international students in similar settings. After a familiarization process, we showed the students randomly arranged talking avatars without voices and asked them to identify the corresponding identities and emotional types of the subjects whose facial expressions were utilized in the creation of the avatars, and to rate their confidence in their selections. We found that the US group had higher success rates in subject identification, although the related difference in confidence ratings between two groups was not significant. The differences in the success rates and confidence ratings on the perception of emotion between the two groups were not significant either. The results of our study provide interesting insights into avatar-based interaction where the national and/or cultural background of a person impacts the perception of identity while having little effect on the perception of emotion. However, we observed that dynamics (e.g., head motion) could offset the disadvantage of cultural unfamiliarity in subject identification. We observed that both groups performed at a nearly identical level in subject identification and emotion perception when they were shown the avatar animation with heightened expression and dynamic intensities. In addition, we observed that the confidence ratings were correlated to accuracy in identifying the subject but not to accuracy in perceiving emotion.

REFERENCES

• 1: Elisabeth André , Thomas Rist , Jochen Müller, Guiding the user through dynamically generated hypermedia presentations with a life-like character, Proceedings of the 3rd international conference on Intelligent user interfaces, p.21-28, January 06-09, 1998, San Francisco, California, United States [doi>10.1145/268389.268394]
• 2: Jeremy N. Bailenson , Nick Yee, A longitudinal study of task performance, head movements, subjective report, simulator sickness, and transformed social interaction in collaborative virtual environments, Presence: Teleoperators and Virtual Environments, v.15 n.6, p.699-716, December 2006 [doi>10.1162/pres.15.6.699]
• 3: Bartneck, C., Takahashi, T., and Katagiri, Y. 2004. Cross-cultural study of expressive avatars. In Proceedings of the Social Intelligence Design, 21--27.
• 4: Christoph Bartneck , Juliane Reichenbach, Subtle emotional expressions of synthetic characters, International Journal of Human-Computer Studies, v.62 n.2, p.179-192, February 2005 [doi>10.1016/j.ijhcs.2004.11.006]
• 5: Beaupre, M. G. and Hess, U. 2005. Cross-cultural emotion perception among Canadian ethnic groups. J. Cross-Cultural Psychology 36, 3, 355--370.
• 6: Bente, G., Kramer, N. C., Peterson, A., and de Ruiter, J. P. 2001. Computer animated movement and person perception: Methodological advances in nonverbal behavior research. J. Nonverbal Behavior 25, 3, 151--166.
• 7: Joseph A. Bonito , Judee K. Burgoon , Bjorn Bengtsson, The role of expectations in human-computer interaction, Proceedings of the international ACM SIGGROUP conference on Supporting group work, p.229-238, November 14-17, 1999, Phoenix, Arizona, United States [doi>10.1145/320297.320324]
• 8: Carlos Busso , Zhigang Deng , Serdar Yildirim , Murtaza Bulut , Chul Min Lee , Abe Kazemzadeh , Sungbok Lee , Ulrich Neumann , Shrikanth Narayanan, Analysis of emotion recognition using facial expressions, speech and multimodal information, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027968]
• 9: Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. F. 2000. Embodied Conversational Agents. MIT Press, Cambridge, MA.
• 10: Deng, Z., Bailenson, J., Lewis, J. P., and Neumann, U. 2006. Perceiving visual emotions with speech. In Proceedings of the 6th International Conference on Intelligence Virtual Agents (IVA) 4133, 107--120.
• 11: Elfenbein, H. A., Levesque, M., Beaupre, M., and Hess, U. 2007. Toward a dialect theory: Cultural differences in the expression and recognition of posed facial expressions. Emotion 7, 1, 131--146.
• 12: Elfenbein, H. A. and Ambady, N. 2003. When familiarity breeds accuracy: Cultural exposure and facial emotion perception. J. Personality Social Psychology 85, 2, 276--290.
• 13: Elfenbein, H. A., Mandal, M. K., Ambady, N., Harizuka, S., and Kumar, S. 2002. Cross-cultural patterns in emotion perception: Highlighting design and analytical techniques. Emotion 2, 1, 75--84.
• 14: Elfenbein, H. A. and Ambady, N. 2002. Is there an in-group advantage in emotion perception? Psychological Bull. 128, 2, 243--249.
• 15: Ekman, P. 1994. Strong evidence for universals in facial expressions: A reply to Russell's mistaken critique. Psychological Bull. 115, 2, 268--287.
• 16: Ekman, P. and Friesen, W. V. 1971. Constants across cultures in the face and emotion. J. Personality Social Psychology 17, 2, 124--129.
• 17: Ekman, P., Friesen, W. V., O'Sullivan, M., Diacoyanni-Tarlatzis, I., Krause, R., Pitcaim, T., Scherer, K., Chan, A., Heider, K., Lccompte, W. A., Ricci-Bitti, P. E., and Tomita, M. 1987. Universal and cultural differences in the judgments of facial expressions of emotion. J. Personality Social Psychology 53, 4, 712--717.
• 18: Fabri, M., Moore, D. J., and Hobbs, D. J. 2002. Expressive agents: Non-verbal communication in collaborative virtual environments. In Proceedings of Autonomous Agents and Multi-Agent Systems (Embodied Conversational Agents).
• 19: Frank, M. G. and Stennett, J. 2001. The forced-choice paradigm and the perception of facial expressions of emotion. J. Personality Social Psychology 80, 1, 75--85.
• 20: Arthur M. Freeman, Creating Emotion in Games: The Art and Craft of Emotioneering, New Riders Publishing, Thousand Oaks, CA, 2003
• 21: Maia Garau , Mel Slater , Simon Bee , Martina Angela Sasse, The impact of eye gaze on communication using humanoid avatars, Proceedings of the SIGCHI conference on Human factors in computing systems, p.309-316, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365121]
• 22: Maia Garau , Mel Slater , Vinoba Vinayagamoorthy , Andrea Brogni , Anthony Steed , M. Angela Sasse, The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642703]
• 23: Jonathan Gratch , Stacy Marsella, Evaluating a Computational Model of Emotion, Autonomous Agents and Multi-Agent Systems, v.11 n.1, p.23-43, July 2005 [doi>10.1007/s10458-005-1081-1]
• 24: Jonathan Gratch , Jeff Rickel , Elisabeth André , Justine Cassell , Eric Petajan , Norman Badler, Creating Interactive Virtual Humans: Some Assembly Required, IEEE Intelligent Systems, v.17 n.4, p.54-63, July 2002 [doi>10.1109/MIS.2002.1024753]
• 25: Guadagno, R. E., Blascovich, J., Bailenson, J. N., and McCall, C. 2007. Virtual humans and persuasion: The Effects of Agency and Behavioral Realism. Media Psychology, 10, 1, 22.
• 26: Sin-Hwa Kang , James H. Watt , Sasi Kanth Ala, Communicators' Perceptions of Social Presence as a Function of Avatar Realism in Small Display Mobile Communication Devices, Proceedings of the Proceedings of the 41st Annual Hawaii International Conference on System Sciences, p.147, January 07-10, 2008 [doi>10.1109/HICSS.2008.95]
• 27: Hess, U., Blairy, S., and Kleck, R. E. 1997. The intensity of emotional facial expressions and decoding accuracy. J. Nonverbal Behavior 21, 4, 241--257.
• 28: Hongpaisanwiwat, C. and Lewis, M. 2003. Attentional effect of animated character. In Proceedings of the Human-Computer Interaction (IFIP INTERACT03), 423--430.
• 29: Katsyri, J. 2006. Human recognition of basic emotions from posed and animated dynamic facial expressions. Ph.D. dissertation, Helsinki University of Technology.
• 30: Katsyri, J., Klucharev, V., Frydrych, M., and Sams, M. 2003. Identification of synthetic and natural emotional facial expression. In Proceedings of the International Conference on Auditory-Visual Speech Processing (AVSP'2003), 239--244.
• 31: Jari Kätsyri , Mikko Sams, The effect of dynamics on identifying basic emotions from synthetic and natural faces, International Journal of Human-Computer Studies, v.66 n.4, p.233-242, April, 2008 [doi>10.1016/j.ijhcs.2007.10.001]
• 32: Kline, C. and Blumberg, B. 1999. The art and science of synthetic character design. In Proceedings of the AISB 1999 Symposium on AI and Creativity in Entertainment and Visual Art.
• 33: Knight, B. and Johnson, A. 1997. The role of movement in face recognition. Visual Cognition 4, 3, 265--273.
• 34: Koda, T. and Maes, P. 1996. Agents with faces: The effect of personification. In Proceedings of the 5th IEEE International Workshop on Robot and Human Communication (RO-MAN'96), 189--194.
• 35: Tomoko Koda , Toru Ishida, Cross-Cultural Study of Avatar Expression Interpretations, Proceedings of the International Symposium on Applications on Internet, p.130-136, January 23-27, 2006 [doi>10.1109/SAINT.2006.19]
• 36: Lander, K., Bruce, V., and Hill, H. 2001. Evaluating the effectiveness of pixelation and blurring on masking theidentity of familiar faces. Applied Cognitive Psychology 15, 1, 101--116.
• 37: Lewis, J. and Purcell, P. 1984. Soft machine: A personable interface. In Proceedings of the Graphics Interface, 223--226.
• 38: Marsella, S. and Gratch, J. 2001. Modeling the interplay of plans and emotions in multi-agent simulations. In Proceedings of the 23rd Annual Conference of the Cognitive Science Society.
• 39: Matsumoto, D. 2007. Emotion judgments do not differ as a function of perceived nationality. Int. J. Psychology 42, 3, 207--214.
• 40: Matsumoto, D. 2002. Methodological requirements to test a possible in-group advantage in judging emotions across the cultures: Comment on Elfenbein and Ambady (2002) and evidence. Psychological Bull. 128, 2, 236--242.
• 41: Clifford Nass , Eun-Young Kim , Eun-Ju Lee, When my face is the interface: an experimental comparison of interacting with one's own face or someone else's face, Proceedings of the SIGCHI conference on Human factors in computing systems, p.148-154, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274667]
• 42: Nowak, K. L. and Rauh, C. 2005. The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. J. Computer-Mediated Communication 11, 1, 153--178.
• 43: Pandzic, I. S., Ostermann, J., and Millen, D. 1999. User evaluation: Synthetic talking faces for interactive services. The Visual Computer 15, 330--340.
• 44: Thomas Rist , Elisabeth André , Jochen Müller, Adding animated presentation agents to the interface, Proceedings of the 2nd international conference on Intelligent user interfaces, p.79-86, January 06-09, 1997, Orlando, Florida, United States [doi>10.1145/238218.238298]
• 45: D. A. Roark , A. J. O'Toole , H. Abdi, Human Recognition of Familiar and Unfamiliar People in Naturalistic Video, Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, p.36, October 17-17, 2003
• 46: Ruttkay, Z., Dormann, C., and Noot, H. 2002. Evaluating ECAs -- What and how? In Proceedings of the AAMAS02 Workshop on Embodied Conversational Agents.
• 47: Lee Sproull , Mani Subramani , Sara Kiesler , Janet H. Walker , Keith Waters, When the interface is a face, Human-Computer Interaction, v.11 n.2, p.97-124, June 1996 [doi>10.1207/s15327051hci1102_1]
• 48: Janet H. Walker , Lee Sproull , R. Subramani, Using a human face in an interface, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.85-91, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191708]
• 49: Nick Yee , Jeremy N Bailenson , Kathryn Rickertsen, A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240626]
• 50: Yuki, M., Maddux, W. W., and Masuda, M. 2007. Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. J. Experimental Social Psychology 43, 303--311.
• 51: Catherine Zanbaka , Paula Goolkasian , Larry Hodges, Can a virtual cat persuade you?: the role of gender and realism in speaker persuasiveness, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124945]
• 52: Zubek, R. and Khoo, A. 2002. Making the human care: On building engaging bots. In Proceedings of the AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Benchmarking

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Human Factors

Keywords:
Subject identification, confidence rating, emotion perception, facial emotion, facial expression, motion capture, user study