MAUI: a multimodal affective user interface

ABSTRACT

Human intelligence is being increasingly redefined to include the all-encompassing effect of emotions upon what used to be considered 'pure reason'. With the recent progress of research in computer vision, speech/prosody recognition, and bio-feedback, real-time recognition of affect will enhance human-computer interaction considerably, as well as assist further progress in the development of new emotion theories.In this article, we describe how affect, moods and emotions closely interact with cognition and how affect and emotion are the quintessential multimodal processes in humans. We then propose an adaptive system architecture designed to sense the user's emotional and affective states via three multimodal subsystems (V, K, A): namely (1) the Visual (from facial images and videos), (2) Kinesthetic (from autonomic nervous system (ANS) signals), and (3) Auditory (from speech). The results of the system sensing are then integrated into the multimodal perceived multimodal anthropomorphic interface agent then adapts its interface by responding most appropriately to the current emotional states of its user, and provides intelligent multi-modal feedback to the user.

REFERENCES

• 1: Wendy S. Ark , D. Christopher Dryer , Davia J. Lu, The Emotion Mouse, Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I, p.818-823, August 22-26, 1999
• 2: Joseph Bates, The role of emotion in believable agents, Communications of the ACM, v.37 n.7, p.122-125, July 1994 [doi>10.1145/176789.176803]
• 3: Birdwhistle. 1970. Kinesics and Context: Essays on Body Motion and Communication. University of Pennsylvania Press.
• 4: Black, and Yacoob, Y. 1995. Recognizing Faces Showing Expressions. In Proceedings of the International Workshop on Automatic Face and Gesture Recognition, IEEE Press.
• 5: M. J. Black , Y. Yacoob, Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion, Proceedings of the Fifth International Conference on Computer Vision, p.374, June 20-23, 1995
• 6: Bower, G. 1981. Mood and Memory. American Psychologist, 36(2).
• 7: Cahn, J. 1990. The Generation of Affect in Synthesized Speech. Journal of the American Voice I/O Society, 8: 1--19.
• 8: Chovil, N. 1991. Discourse-Oriented Facial Displays in Conversation. Research on Language and Social Interaction 25: 163--194.
• 9: Garrison W. Cottrell , Janet Metcalfe, EMPATH: face, emotion, and gender recognition using holons, Proceedings of the 1990 conference on Advances in neural information processing systems 3, p.564-571, October 1990, Denver, Colorado, United States
• 10: Martha E. Crosby , Brent Auernheimer , Christoph Aschwanden , Curtis Ikehara, Physiological data feedback for application in distance education, Proceedings of the 2001 workshop on Perceptive user interfaces, November 15-16, 2001, Orlando, Florida [doi>10.1145/971478.971496]
• 11: Damasio, A. 1994. Descartes' Error, New York: Avon Books.
• 12: Derryberry, D. and Tucker, D. 1992. Neural Mechanisms of Emotion. Journal of Consulting and Clinical Psychology 60(3): 329--337.
• 13: Dyer, M. 1987. Emotions and their Computations: Three Computer Models. Cognition and Emotion, 1(3): 323--347.
• 14: Ekman, P., and Friesen, W. V. 1975. Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Englewood Cliffs, New Jersey: Prentice Hall, Inc.
• 15: Elliott, C. 1994. Components of Two-way Emotion Communication between Humans and Computers Using Broad, Rudimentary Model of Affect and Personality. Cognitive Studies: Bulletin of the Japanese Cognitive Society, 1(2): 16--30.
• 16: Essa, I., Darrell, T. and Pentland, A. 1994. Tracking Facial Motion. In Proceedings of the IEEE Workshop on Nonrigid and Articulate Motion 1994.
• 17: Michael Kenyatta Fleming , David Rumelhart, A neural network model of micro- and macroprosody, 1997
• 18: Frijda, N. and Swagerman J. 1987. Can Computers Feel? Theory and Design of an Emotional System. Cognition and Emotion, 1(3): 235--257.
• 19: Frijda, N. 1986. The Emotions. New-York: Cambridge University Press.
• 20: Goleman, D. 1995. Emotional Intelligence. New-York: Bantam Books.
• 21: Healey J. and Picard R. 2000. SmartCar: Detecting Driver Stress. In Proceedings of ICPR'00, Barcelona, Spain, 2000.
• 22: Jameson, A., Cecile Paris and Tasso, C. 1997. User Modeling: Proceedings of, UM'97, New-York: Springer Wien New York.
• 23: Kearney, G. and McKenzie, S. 1993. Machine Interpretation of Emotion: Design of a Memory-Based Expert System for Interpreting Facial Expressions in Terms of Signaled Emotions, Cognitive Science 17.
• 24: Ledoux, J. 1992. Brain Mechanisms of Emotion and Emotional Learning. Current Opinion in Neurobiology 2: 191--197.
• 25: Barbara Hayes-Roth , Gene Ball , Christine Lisetti , Rosalind W. Picard , Andrew Stern, Panel on affect and emotion in the user interface, Proceedings of the 3rd international conference on Intelligent user interfaces, p.91-94, January 06-09, 1998, San Francisco, California, United States [doi>10.1145/268389.268406]
• 26: Christine L. Lisetti, Motives for intelligent agents: computational scripts for emotion concepts, Proceedings of the sixth Scandinavian conference on Artificial intelligence, p.59-70, August 1998, Helsinki, Finland
• 27: Lisetti, C. and Rumelhart, D. 1997b. A Connectionist Model of Emotional Arousal and Regulation. Technical Report, PDP-97-01. Dept. of Psychology. Stanford Parallel Distributed Processing Research Group, Stanford University.
• 28: Pattie Maes, Agents that reduce work and information overload, Communications of the ACM, v.37 n.7, p.30-40, July 1994 [doi>10.1145/176789.176792]
• 29: Murray, I. and Arnott, J. 1993. Toward the Simulation of Emotion in Synthetic Speech: A Review of the Litterature on Human Vocal Emotion. Journal Acoustical Society of America, 93(2): 1097--1108.
• 30: Katashi Nagao , Akikazu Takeuchi, Speech dialogue with facial displays: multimodal human-computer conversation, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.102-109, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981747]
• 31: O'Rorke, P. and Ortony, A. 1994. Explaining Emotions Cognitive Science 18(2): 283--323.
• 32: Pecchinenda, A. and Smith, C. 1996. Affective Significance of Skin Conductance Activity During Difficult Problem-solving Task. Cognition and Emotion, 10(5), 481--503.
• 33: Planalp, S., DeFrancisco, V. and Rutherford, D. 1996. Varieties of Cues to Emotion in Naturally Occurring Situations. Cognition and Emotion, 10(2), 137--153.
• 34: Sinha, R. and Parsons, O. 1996. Multivariate Response Patterning of Fear and Anger. Cognition and Emotion, 10 (2), 173--198. Vrana, S. The Psychophysiology of Disgust: Differentiating Negative Emotional Contexts with Facial EMG. Psychophysiology, 30, 279--286, 1993. X
• 35: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 36: Rosenblum, M., and Yacoob, Y., and Davis, L. 1994. Human Emotion Recognition from Motion Using a Radial Basis Function Network Architecture. In IEEE Workshop on Motion of Non-Rigid and Articulated Objects.
• 37: Henry A. Rowley , Shumeet Baluja , Takeo Kanade, Neural Network-Based Face Detection, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.203, June 18-20, 1996
• 38: Rumelhart, D. 1995. Neuro-Modulation: A Connectionist Approach. In H. Morowitz and J. Singer (Eds.), The Mind, the Brain, and Complex Associative Systems. 145--153. Readings, MA: Addison-Wesley.
• 39: Sloman, A. Motives, Mechanisms, and Emotions. 1990. In Boden, M. Ed., The Philosophy of Artificial Intelligence. New York, NY: Oxford University Press.
• 40: Stonham, T. J. 1986. Practical face recognition and verification with Wisard. In H. Ellis & M.A. Jeeves (Eds.), Aspects of face processing. Lancaster, England: Martinus Nijhoff.
• 41: Terzopoulos, D. and Waters, K. 1990. Analysis and Synthesis of Facial Images Using Physical and Anatomical Models. In Proceedings of the Internationa Conference on Computer Vision, 727--732.
• 42: Turk, M., and Pentland, A. 1991. Eigenfaces for Recognition. Journal of Cognitive Neuroscience 3(1): 71--86.
• 43: Valentin, T., and Abdi, H., and O'Toole, A., and Cottrell, G. 1994. Connectionist Models of Face Processing: A Survey. Pattern Recognition 27: 1290--1230.
• 44: Vrana, S. 1993. The Psychophysiology of Disgust: Differentiating Negative Emotional Contexts with Facial EMG. Psychophysiology, 30, 279--286.
• 45: Wierzbicka, A. 1992. Defining Emotion Concepts. Cognitive Science 16: 539--581.
• 46: Yacoob, Y., and Lam, H., and Davis, L. 1995. Recognizing Faces Showing Expressions. In Proceedings of the International Workshop on Automatic Face and Gesture Recognition, IEEE Press.
• 47: Zajonc, R. 1984. On the Primacy of Affect. American Psychologist 39: 117--124.
• 48: Zajonc, R. and Markus, H. 1984b. Affect and Cognition: the Hard Interface. In Izard, C. and Kagan, J. and Zajonc, R. Ed., Emotion, Cognition and Behavior. Cambridge, MA: Cambridge University Press.
• 49: Zajonc, R. 1989. Feeling and Facial Efference: Implications of the Vascular Theory of Emotion. Psychological Review 39: 117--124.

INDEX TERMS

Keywords:
affect recognition, emotions, intelligent interfaces, interface agent