Exploiting emotions to disambiguate dialogue acts

ABSTRACT

This paper describes an attempt to reveal the user's intention from dialogue acts, thereby improving the effectiveness of natural interfaces to pedagogical agents. It focuses on cases where the intention is unclear from the dialogue context or utterance structure, but where the intention may still be identified using the emotional state of the user. The recognition of emotions is based on physiological user input. Our initial user study gave promising results that support our hypothesis that physiological evidence of emotions could be used to disambiguate dialogue acts. This paper presents our approach to the integration of natural language and emotions as well as our first empirical results, which may be used to endow interactive agents with emotional capabilities.

REFERENCES

• 1: Gregory Aist , Barry Kort , Rob Reilly , Jack Mostow , Rosalind W. Picard, Adding Human-Provided Emotional Scaffolding to an Automated Reading Tutor That Listens Increases Student Persistence, Proceedings of the 6th International Conference on Intelligent Tutoring Systems, p.992, June 02-07, 2002
• 2: Elisabeth André , Martin Klesen , Patrick Gebhard , Steve Allen , Thomas Rist, Integrating models of personality and emotions into lifelike characters, Affective interactions: towards a new generation of computer interfaces, Springer-Verlag New York, Inc., New York, NY, 2001
• 3: G. Ball and J. Breese. Modeling the emotional state of computer users. In Workshop on Attitude, Personality and Emotions in User-Adapted Interaction, 1999.
• 4: N. Chomsky. On certain formal properties of grammars. Information and Control, 2(2):137--167, 1959.
• 5: C. Conati, R. Chabbal, and H. Maclaren. A study on using biometric sensors for detecting user emotions in educational games. In 3rd Workshop on Affective and Attitude User Modeling, Pittsburgh, USA, June 2003.
• 6: Cristina Conati , Abigail Gertner , Kurt Vanlehn, Using Bayesian Networks to Manage Uncertainty in Student Modeling, User Modeling and User-Adapted Interaction, v.12 n.4, p.371-417, November 2002 [doi>10.1023/A:1021258506583]
• 7: R. Cowie, E. Douglas-Cowie, N. Tsapsoulis, G. Votsis, S. Kollias, W. A. Fellens, and J. G. Taylor. Emotion recognition in human-computer interaction, 2001.
• 8: T. Dutoit, V. Pagel, N. Pierret, O. van~der Vreken, and F. Bataille. The mbrola project: Towards a set of high-quality speech synthesizers free of use for non-commercial purposes. In Proceedings of the Fourth International Conference on Spoken Language Processing, Philadelphia, USA, October 1996.
• 9: C. Elliott, J. C. Lester, and J. Rickel. Integrating affective computing into animated tutoring agents. In Proceedings of the IJCAI Workshop on Animated Interface Agents: Making Them Intelligent, pages 113--121, Nagoya, Japan, 1997.
• 10: J. Healey and R. W. Picard. Digital processing of affective signals. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, Seattle, Washington, May 1997.
• 11: T. S. Huang, L. S. Chen, H. Tao, T. Miyasato, and R. Nakatsu. Bimodal emotion recognition by man and machine. In ATR Workshop on Virtual Communication Environments - Bridges over Art/Kansei and VR Technologies, Kyoto, Japan, April 1998.
• 12: Finn V. Jensen, Bayesian Networks and Decision Graphs, Springer-Verlag New York, Inc., Secaucus, NJ, 2001
• 13: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 14: P. J. Lang. The emotion probe: Studies of motivation and attention. American Psychologist, 50(5):372--385, 1995.
• 15: A. Langenmayr. Sprachpsychologie. Hogrefe, Göttingen, 1997.
• 16: Hugo Liu , Henry Lieberman , Ted Selker, A model of textual affect sensing using real-world knowledge, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604067]
• 17: Stacy C. Marsella , W. Lewis Johnson , Catherine LaBore, Interactive pedagogical drama, Proceedings of the fourth international conference on Autonomous agents, p.301-308, June 03-07, 2000, Barcelona, Spain [doi>10.1145/336595.337507]
• 18: B. Martinovsky and D. Traum. Breakdown in human-machine interaction: the error is the clue. In Proceedings of the ISCA tutorial and research workshop on Error handling in dialogue systems, pages 11--16, August 2003.
• 19: Mehryar Mohri, Finite-state transducers in language and speech processing, Computational Linguistics, v.23 n.2, p.269-311, June 1997
• 20: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 21: R. W. Picard, Toward computers that recognize and respond to user emotion, IBM Systems Journal, v.39 n.3-4, p.705-719, July 2000
• 22: R. W. Picard and J. Healey. Affective wearables. Personal Technologies, 1(4):231--240, 1997.
• 23: R. W. Picard and J. Klein. Computers that recognise and respond to user emotion: Theoretical and practical implications. Interacting with Computers, 14(2):141--169, 2002.
• 24: Byron Reeves , Clifford Nass, The media equation: how people treat computers, television, and new media like real people and places, Cambridge University Press, New York, NY, 1996
• 25: Carolyn P. Rosé , Diane Litman , Dumisizwe Bhembe , Kate Forbes , Scott Silliman , Ramesh Srivastava , Kurt VanLehn, A comparison of tutor and student behavior in speech versus text based tutoring, Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing, p.30-37, May 31-31, 2003 [doi>10.3115/1118894.1118899]
• 26: R. Schandry. Lehrbuch Psychophysiologie. Psychologie Verlags Union, studienausgabe edition, 1998.
• 27: J. Scheirer, R. Fernandez, J. Klein, and R. W. Picard. Frustrating the user on purpose: A step toward building an affective computer. Interacting with Computers, 14(2):93--118, 2002.
• 28: Thomas A. Sudkamp, Languages and machines: an introduction to the theory of computer science, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 29: E. Vyzas and R. W. Picard. Affective pattern classification. In D. Canamero, editor, Emotional and Intelligent: The Tangled Knot of Cognition, pages 176--182, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Human Factors

Keywords:
affective user interfaces, dialogue acts, multimodal integration, natural language processing, pedagogical discourse