Embodiment in conversational interfaces: Rea

ABSTRACT

In this paper, we argue for embodied corrversational characters as the logical extension of the metaphor of human - computer interaction as a conversation. We argue that the only way to fully model the richness of human I&+ to-face communication is to rely on conversational analysis that describes sets of conversational behaviors as fi~lfilling conversational functions, both interactional and propositional. We demonstrate how to implement this approach in Rea, an embodied conversational agent that is capable of both multimodal input understanding and output generation in a limited application domain. Rea supports both social and task-oriented dialogue. We discuss issues that need to be addressed in creating embodied conversational agents, and describe the architecture of the Rea interface.

REFERENCES

• 1: Elisabeth André , Thomas Rist , Jochen Müller, Integrating reactive and scripted behaviors in a life-like presentation agent, Proceedings of the second international conference on Autonomous agents, p.261-268, May 10-13, 1998, Minneapolis, Minnesota, United States [doi>10.1145/280765.280842]
• 2: Azarbayejani, A., Wren, C. and Pentland A. Real-time 3-D tracking of the human body. In Proceedings of 1MAGE'COM 96, (Bordeaux, France, May 1996).
• 3: Gene Ball , Dan Ling , David Kurlander , John Miller , David Pugh , Tim Skelly , Andy Stankosky , David Thiel , Maarten Van Dantzich , Trace Wax, Lifelike computer characters: the persona project at Microsoft, Software agents, MIT Press, Cambridge, MA, 1997
• 4: Beskow, J. and McGlashan, S.Olga - A Conversational Agent with Gestures, In Proceedings of the IJCAI'97 workshop on Animated Interface Agents - Making them intelligent, (Nagoya, Japan, August 1997), Morgan-Kaufmann Publishers, San Francisco.
• 5: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 6: Richard A. Bolt , Edward Herranz, Two-handed gesture in multi-modal natural dialog, Proceedings of the 5th annual ACM symposium on User interface software and technology, p.7-14, November 15-18, 1992, Monteray, California, United States [doi>10.1145/142621.142623]
• 7: Brennan, S.E. and Hulteen, E.A. Interaction and Feedback in a Spoken Language System. Knowledge- Based Systems, 8,2 (April-June 1995), 143-151.
• 8: Justine Cassell , Catherine Pelachaud , Norman Badler , Mark Steedman , Brett Achorn , Tripp Becket , Brett Douville , Scott Prevost , Matthew Stone, Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.413-420, July 1994 [doi>10.1145/192161.192272]
• 9: Cassell, J. and Th6risson, K. The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Conversational Agents. Journal of Applied Artificial Intelligence, in press.
• 10: Cassell, J., Torres, O. and Prevost, S. Turn taking vs. Discourse Structure: how best to model multimodal conversation. In Wilks (ed.) Machine Conversations. Kluwer, The Hague, 1998.
• 11: Clark, H.H. and Brennan, S.E. Grounding in Communication. In Shared Cognition: Thinking as Social Practice, J. Levine, L.B. Resnick and S.D. Behrend, (eds.). APA Books, Washington, D.C, 1991.
• 12: CLIPS Reference Manual Version 6.0. Technical Report, Number JSC-25012, Software Technology Branch, Lyndon B. Johnson Space Center, Houston, TX, 1994.
• 13: Tim Finin , Richard Fritzson , Don McKay , Robin McEntire, KQML as an agent communication language, Proceedings of the third international conference on Information and knowledge management, p.456-463, November 29-December 02, 1994, Gaithersburg, Maryland, United States [doi>10.1145/191246.191322]
• 14: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 15: Kendon, A. The negotiation of context in face-to-face interaction. In A. Duranti and C. Goodwin (eds.), Rethinking context: language as interactive phenomenon. Cambridge University Press. NY, 1990.
• 16: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 17: A. Bryan Loyall , Joseph Bates, Personality-rich believable agents that use language, Proceedings of the first international conference on Autonomous agents, p.106-113, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267681]
• 18: Katashi Nagao , Akikazu Takeuchi, Social interaction: multimodal conversation with social agents, Proceedings of the twelfth national conference on Artificial intelligence (vol. 1), p.22-28, October 1994, Seattle, Washington, United States
• 19: R. S. Nickerson, On conversational interaction with computers, Proceedings of the ACM/SIGGRAPH workshop on User-oriented design of interactive graphics systems, October 14-15, 1976, Pittsburgh, PA [doi>10.1145/1024273.1024286]
• 20: Noma, T. and Badler, N. (1997). A virtual human presenter. In Proceedings of the IJCAI'97 workshop on Animated interface Agents - Making them Intelligent, (Nagoya, Japan, August, 1997), Morgan-Kaufrnann Publishers, San Francisco.
• 21: Sharon Oviatt, User-Centered Modeling for Spoken Language and Multimodal Interfaces, IEEE MultiMedia, v.3 n.4, p.26-35, December 1996 [doi>10.1109/93.556458]
• 22: Byron Reeves , Clifford Nass, The media equation: how people treat computers, television, and new media like real people and places, Cambridge University Press, New York, NY, 1996
• 23: Sparrell, C. J. Coverbal Iconic Gestures in Human- Computer Interaction. S.M. Thesis, MIT Media Arts and Sciences Section, 1993.
• 24: Steedman, M. Structure and intonation. Language, 1991, 67(2), 190-296.
• 25: Akikazu Takeuchi , Katashi Nagao, Communicative facial displays as a new conversational modality, Proceedings of the SIGCHI conference on Human factors in computing systems, p.187-193, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/169059.169156]
• 26: Th6risson, K. R. Communicative Humanoids: A Computational Model of Psychosocial Dialogue Skills. PhD Thesis, MIT Media Laboratory, 1996.
• 27: Wolfgang Wahlster , Elisabeth André , Winfried Graf , Thomas Rist, Designing illustrated texts: how language production is influenced by graphics generation, Proceedings of the fifth conference on European chapter of the Association for Computational Linguistics, p.8-14, April 09-11, 1991, Berlin, Germany [doi>10.3115/977180.977183]
• 28: Joseph Weizenbaum, ELIZA—a computer program for the study of natural language communication between man and machine, Communications of the ACM, v.9 n.1, p.36-45, Jan. 1966 [doi>10.1145/365153.365168]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

General Terms:
Experimentation, Human Factors

Keywords:
conversational characters, intelligent agents, multimodal input, multimodal output