Affective multimodal human-computer interaction

ABSTRACT

Social and emotional intelligence are aspects of human intelligence that have been argued to be better predictors than IQ for measuring aspects of success in life, especially in social interactions, learning, and adapting to what is important. When it comes to machines, not all of them will need such skills. Yet to have machines like computers, broadcast systems, and cars, capable of adapting to their users and of anticipating their wishes, endowing them with the ability to recognize user's affective states is necessary. This article discusses the components of human affect, how they might be integrated into computers, and how far are we from realizing affective multimodal human-computer interaction.

REFERENCES

• 1: Ambady, N. and Rosenthal, R. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111, 2 (Feb. 1992), 256--274.
• 2: Balomenos, T., Raouzaiou, A., Ioannou, S., Drosopoulos, A., Karpouzis, K. and Kollias, S. Emotion Analysis in Man-Machine Interaction Systems. Machine Learning for Multimodal Interaction, Lecture Notes in Computer Science, vol. 3361, Bengio, S. and Bourlard, H., Eds. Springer-Verlag, Berlin, D, 2005, 318--328.
• 3: Banse, R., & Scherer, K. R. Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 1996, 614--636.
• 4: Chen, L.S. and Huang, T.S. Emotional expressions in audiovisual human computer interaction. In Proc. Int'l Conf. Multimedia and Expo, 2000, 423--426.
• 5: Lawrence Shao-Hsien Chen , Thomas S. Huang, Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction, 2000
• 6: Chen, T. and Rao, R.R., Audio-visual integration in multimodal communication, Proceedings of the IEEE, 86, 5 (May 1998), 837--852.
• 7: Ira Cohen , Nicu Sebe , Ashutosh Garg , Lawrence S. Chen , Thomas S. Huang, Facial expression recognition from video sequences: temporal and static modeling, Computer Vision and Image Understanding, v.91 n.1-2, p.160-187, July 2003 [doi>10.1016/S1077-3142(03)00081-X]
• 8: Jeffrey F. Cohn , Gary S. Katz, Bimodal expression of emotion by face and voice, Proceedings of the sixth ACM international conference on Multimedia: Face/gesture recognition and their applications, p.41-44, September 1998, Bristol, United Kingdom [doi>10.1145/306668.306683]
• 9: Cohn, J. F. and Schmidt, K. L., The timing of facial motion in posed and spontaneous smiles. Wavelets, Multiresolution and Information Processing, 2, 2004, 1--12.
• 10: Liyanage C. DeSilva , Pei Chi Ng, Bimodal Emotion Recognition, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.332, March 26-30, 2000
• 11: Ekman, P. and Friesen, W.F. The repertoire of nonverbal behavioral categories - origins, usage, and coding. Semiotica, 1, 1969, 49--98.
• 12: Frank, M.G., Ekman, P. and Friesen, W., Behavioral markers and recognizability of the smile of enjoyment. Journal of Personality and Social Psychology, 64, 1 (Jan. 1993), 83--93.
• 13: Fridlund, A.J. The new ethology of human facial expression. The psychology of facial expression. Russell, J.A. and Fernandez-Dols, J.M., Eds. Cambridge University Press, Cambridge, MA, USA, 1997, 103--129.
• 14: Garg, A., Naphade, M., Huang, T.S. Modeling video using input/output Markov models with application to multimodal event detection. Handbook of Video Databases: Design and Applications, B. Furth, O. Marques, and B. Furth, Eds., 2003.
• 15: Ashutosh Garg , Vladimir Pavlovic , James M. Rehg, Audio-Visual Speaker Detection Using Dynamic Bayesian Networks, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.384, March 26-30, 2000
• 16: Garg, A., Pavlovic, V., Rehg, J. Boosted learning in dynamic Bayesian networks for multimodal speaker detection, Proceedings of the IEEE, 91, 9 (Sep. 2003), 1355--1369.
• 17: Go, H.J., Kwak, K.C., Lee, D.J. and Chun, M.G. Emotion recognition from facial image and speech signal. In Proc. Conf. of the Society of Instrument and Control Engineers, 2003, 2890--2895.
• 18: Goleman, D. Emotional Intelligence. Bantam Books, New York, NY, USA, 1995.
• 19: Hu, W., Tan, T., Wang, L., Maybank, S. A survey on visual surveillance of object motion and behaviors, IEEE Trans. On Systems, Man, and Cybernetics - Part C: Applications and Reviews, 34, 3 (Aug. 2004), 334--352.
• 20: Donna L. Hoffman , Thomas P. Novak , Alladi Venkatesh, Has the Internet become indispensable?, Communications of the ACM, v.47 n.7, p.37-42, July 2004 [doi>10.1145/1005817.1005818]
• 21: Juslin, P.N. and Scherer, K.R. Vocal expression of affect. In The New Handbook of Methods in Nonverbal Behavior Research. Harrigan, J., Rosenthal, R. and Scherer, K., Eds. Oxford University Press, Oxford, UK, 2005.
• 22: Takeo Kanade , Yingli Tian , Jeffrey F. Cohn, Comprehensive Database for Facial Expression Analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.46, March 26-30, 2000
• 23: Keltner, D. and Ekman, P. Facial expression of emotion. Handbook of Emotions. Lewis, M. and Haviland-Jones, J.M., Eds. Guilford Press, New York, NY, USA, 2000, 236--249.
• 24: Larsen, R.J. and Diener, E. Promises and problems with the circumplex model of emotion. Emotion, vol. 13, Review of Personality and Social Psychology, M. S. Clark, Ed., Sage Publications, Newbury Park, USA, 1992, 25--59.
• 25: Matsumoto, D. Cultural similarities and differences in display rules. Motivation and Emotion, 14, 1990, 195--214.
• 26: Donald A. Norman, Human-centered design considered harmful, interactions, v.12 n.4, July + August 2005 [doi>10.1145/1070960.1070976]
• 27: Nwe, T.L., Wei, F.S. and De Silva, L.C., Speaker Dependent Emotional Speech Recognition Using Hidden Markov Models. Speech Communications, 41, 4 (Nov. 2003), 603--623.
• 28: Oviatt, Süser-centered modeling and evaluation of multimodal interfaces. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1457--1468.
• 29: Pan, H., Liang, Z.P., Anastasio, T.J., Huang, T.S., Exploiting the dependencies in information fusion, In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, 407--412, 1999.
• 30: Pantic, M. and Rothkrantz, L.J.M. Toward an Affect-Sensitive Multimodal Human-Computer Interaction. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1370--1390.
• 31: Pantic, M. and Rothkrantz, L.J.M. Case-based reasoning for user-profiled recognition of emotions from face images. In Proc. Int'l Conf. Multimedia and Expo, 2004, 391--394.
• 32: Pantic, M., Valstar, M.F., Rademaker, R. and Maat, L. Web-based database for facial expression analysis. In Proc. Int'l Conf. Multimedia and Expo, 2005. (www.mmifacedb.com)
• 33: Catherine Pelachaud , Valeria Carofiglio , Berardina De Carolis , Fiorella de Rosis , Isabella Poggi, Embodied contextual agent in information delivering application, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2, July 15-19, 2002, Bologna, Italy [doi>10.1145/544862.544921]
• 34: Alex Pentland, Looking at People: Sensing for Ubiquitous and Wearable Computing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.1, p.107-119, January 2000 [doi>10.1109/34.824823]
• 35: Alex (Sandy) Pentland, Socially Aware Computation and Communication, Computer, v.38 n.3, p.33-40, March 2005 [doi>10.1109/MC.2005.104]
• 36: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 37: Rosalind W. Picard , Elias Vyzas , Jennifer Healey, Toward Machine Emotional Intelligence: Analysis of Affective Physiological State, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.10, p.1175-1191, October 2001 [doi>10.1109/34.954607]
• 38: Russell, J.A. Is there universal recognition of emotion from facial expression? Psychological Bulletin, 115, 1 (Jan. 1994), 102--141.
• 39: Scanlon, P. and Reilly, R.B. Feature analysis for automatic speech reading. In Proc. Int'l Workshop Multimedia Signal Processing, 2001, 625--630.
• 40: Scherer, K.R. and Ekman, P., Eds., Handbook of methods in non-verbal behavior research. Cambridge University Press, Cambridge, USA, 1982.
• 41: Sebe. N, Lew, M.S., Cohen, I.,Sun, Y., Gevers, T., Huang, T.S., Authentic facial expression analysis. In Proc. Int'l Conf. Face and Gesture Recognition, 2004, 517--522.
• 42: Song, M., Bu, J., Chen, C. and Li, N. Audio-visual based emotion recognition - A new approach. In Proc. Int'l Conf. Computer Vision and Pattern Recognition, 2004, 1020--1025.
• 43: Watson, D., Clark, L.A., Weber, K., Smith-Assenheimer, J., Strauss, M.E. and McCormick, R.A. Testing a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult, and patient samples. Journal of Abnormal Psychology, 104, (Jan 1995), 15--25.
• 44: Watson, D., Weber, K., Assenheimer, J.S., Clark, L.A., Strauss, M. E. and McCormick, R.A. Testing a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression symptom scales. Journal of Abnormal Psychology, 104, (Jan 1995), 3--14.
• 45: Wierzbicka, A. Reading human faces. Pragmatics and Cognition, 1, 1 (Jan. 1993), 1--23.
• 46: Yoshitomi, Y., Kim, S., Kawano, T. and Kitazoe, T. Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In Proc. Int'l Workshop on Robot-Human, 2000, 178--183.
• 47: JAFFE: www.mic.atr.co.jp/~mlyons/jaffe.html

INDEX TERMS

Primary Classification:
  A. General Literature
  A.1 INTRODUCTORY AND SURVEY

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications

General Terms:
Algorithms, Performance, Theory

Keywords:
affective computing, multimodal human-computer interaction