Multimodal transformed social interaction

ABSTRACT

Understanding human-human interaction is fundamental to the long-term pursuit of powerful and natural multimodal interfaces. Nonverbal communication, including body posture, gesture, facial expression, and eye gaze, is an important aspect of human-human interaction. We introduce a paradigm for studying multimodal and nonverbal communication in collaborative virtual environments (CVEs) called Transformed Social Interaction (TSI), in which a user's visual representation is rendered in a way that strategically filters selected communication behaviors in order to change the nature of a social interaction. To achieve this, TSI must employ technology to detect, recognize, and manipulate behaviors of interest, such as facial expressions, gestures, and eye gaze. In [13] we presented a TSI experiment called <i>non-zero-sum gaze</i> (NZSG) to determine the effect of manipulated eye gaze on persuasion in a small group setting. Eye gaze was manipulated so that each participant in a three-person CVE received eye gaze from a presenter that was normal, less than normal, or greater than normal. We review this experiment and discuss the implications of TSI for multimodal interfaces.

REFERENCES

• 1: Argyle, M. Bodily Communication (2nd edition). London, England, UK: Methuen, 1988.
• 2: Depaulo, B. M., and Friedman, H. S. Nonverbal communication. In D. T. Gilbert, S. T. Fiske, and G. Lindzey (eds.), The Handbook of Social Psychology (4th edition), pp. 3--40, Vol. 2, Boston: McGraw-Hill, 1998.
• 3: Francis Quek , David McNeill , Robert Bryll , Susan Duncan , Xin-Feng Ma , Cemil Kirbas , Karl E. McCullough , Rashid Ansari, Multimodal human discourse: gesture and speech, ACM Transactions on Computer-Human Interaction (TOCHI), v.9 n.3, p.171-193, September 2002 [doi>10.1145/568513.568514]
• 4: Oviatt, S. L. Multimodal interface research: A science without borders. Proceedings of the International Conference of Spoken Language Processing (ICSLP'2000), ed. by B. Yuan, T. Huang & X. Tang, Beijing: Chinese Friendship Publishers, 2000, vol. 3, 1--6.
• 5: Xiaolong Zhang , George W. Furnas, Social interactions in multiscale CVEs, Proceedings of the 4th international conference on Collaborative virtual environments, p.31-38, September 30-October 02, 2002, Bonn, Germany [doi>10.1145/571878.571884]
• 6: Normand, V. et al. The COVEN project: exploring applicative, technical and usage dimensions of collaborative virtual environment. Presence: Teleoperators and Virtual Environments, 8(2), 1999.
• 7: Bailenson, J., Beall, A., Blascovich, J., Loomis, J. and Turk, M. Transformed social interaction: decoupling representation from behavior and form in collaborative virtual environments. PRESENCE: Teleoperators and Virtual Environments, MIT press, Vol. 13, No. 4, August 2004.
• 8: Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., and Bailenson, J. N. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychological Inquiry, 13, 146--149, 2001.
• 9: Beall, A.C., Bailenson, J.N., Loomis, J., Blascovich, J., and Rex, C. Non-Zero-Sum Mutual Gaze in Collaborative Virtual Environments. Proceedings of HCI International, Crete, 2003.
• 10: Cappella, J. N. and Pelachaud, C. Rules for responsive robots: Using human interactions to build virtual interactions. In Vangelisti, A.L., Reis, H.T., and Fitzpatrick, M. (Eds.), Stability and Change in Relationships, pp. 325--354, Cambridge, MA: Cambridge University Press, 2002.
• 11: Morton, G. Effect of eye contact and distance on the verbal reinforcement of attitude. Journal of Social Psychology, 111, pp. 73--78, 1980.
• 12: Sherwood, J. V. Facilitative effects of gaze upon learning. Perceptual and Motor Skills, 64, pp. 1275--1278, 1987.
• 13: Bailenson, J., Beall, A., Blascovich, J., Loomis, J., and Turk, M. Non-zero-sum gaze and persuasion in immersive virtual environments. Manuscript under review.
• 14: Gigorovici, G. Persuasive effects of presence in immersive virtual environments. In G. Riva, F. Davide (Eds), Being There: concepts effects and measurement of user presence in synthetic environments, Amsterdam, The Netherlands: IOS Press, online, 2003.
• 15: Guadagno, R. E. and Cialdini, R. B. Online persuasion: An examination of gender differences in computer-mediated interpersonal influence. Group Dynamics: Theory Research and Practice. Special Issue on Internet Research, 6, pp. 38--51, 2002.
• 16: Lanier, J. Virtually there. Scientific American, April, 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

General Terms:
Experimentation, Human Factors

Keywords:
computer-mediated communication, multimodal processing, transformed social interaction