Bimodal HCI-related affect recognition

ABSTRACT

Perhaps the most fundamental application of affective computing will be Human-Computer Interaction (HCI) in which the computer should have the ability to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

REFERENCES

• 1: Pantic M., Rothkrantz, L.J.M., Toward an affect-sensitive multimodal human-computer interaction, Proceedings of the IEEE, Vol. 91, No. 9, Sept. 2003, 1370--1390
• 2: Chen, L. and Huang, T. S., Emotional expressions in audiovisual human computer interaction, Int. Conf. on Multimedia & Expo 2000, 423--426
• 3: L. S. Chen , T. S. Huang , T. Miyasato , R. Nakatsu, Multimodal Human Emotion/Expression Recognition, Proceedings of the 3rd. International Conference on Face & Gesture Recognition, p.366, April 14-16, 1998
• 4: Liyanage C. DeSilva , Pei Chi Ng, Bimodal Emotion Recognition, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.332, March 26-30, 2000
• 5: Yoshitomi, Y., Kim, S., Kawano, T., and Kitazoe, T., Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face, in Proc. ROMAN 2000, 178--183
• 6: Dan Roth, Learning to resolve natural language ambiguities: a unified approach, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.806-813, July 1998, Madison, Wisconsin, United States
• 7: Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., and Taylor, J.G., Emotion Recognition in Human-Computer Interaction, IEEE Signal Processing Magazine, January 2001, 32--80
• 8: Steeneken, H.J.M. and Hansen, J.H.L., Speech Under Stress Conditions: Overview of the Effect on Speech Production and on System Performance, in Proc. ICASSP, vol. 4, 1999, 2079--2082
• 9: Carlson, A.J., Cumby, C.M., Rizzolo, N.D., Rosen, J.L., and Roth, D., SNoW User Manual, UIUC Tech report UIUC-DCS-R-99-210
• 10: Tu, J., Zhang, Z., Zeng, Z. and Huang, T.S., Face Localization via Hierarchical Condensation with Fisher Boosting Feature Selection, In Proc. Computer Vision and Pattern Recognition, 2004.
• 11: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 12: Mehrabian, A., Communication without words, Psychol. Today, vol.2, no.4, 53--56, 1968
• 13: Sebe, N., Lew, M., Cohen, I., Sun, Y., Gevers, T., and Huang, T.S., Authentic Facial Expression Analysis, Int. Conf. on Automatic Face & Gesture Recognition 2004.
• 14: itr.beckman.uiuc.edu
• 15: Lawrence Shao-Hsien Chen , Thomas S. Huang, Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction, 2000
• 16: H. Tao , T. S. Huang, Connected Vibrations: A Modal Analysis Approach for Non-Rigid Motion Tracking, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.735, June 23-25, 1998

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Collaborative learning

General Terms:
Design, Performance

Keywords:
affect recognition, affective computing, emotion recognition, multimodal human-computer interaction