Foundations of human computing: facial expression and emotion

ABSTRACT

Many people believe that emotions and subjective feelings are one and the same and that a goal of human-centered computing is emotion recognition. The first belief is outdated; the second mistaken. For human-centered computing to succeed, a different way of thinking is needed.Emotions are species-typical patterns that evolved because of their value in addressing fundamental life tasks[19]. Emotions consist of multiple components that may include intentions, action tendencies, appraisals, other cognitions, central and peripheral changes in physiology, and subjective feelings. Emotions are not directly observable, but are inferred from expressive behavior, self-report, physiological indicators, and context. I focus on expressive behavior because of its coherence with other indicators and the depth of research on the facial expression of emotion in behavioral and computer science. In this paper, among the topics I include are approaches to measurement, timing or dynamics, individual differences, dyadic interaction, and inference. I propose that design and implementation of perceptual user interfaces may be better informed by considering the complexity of emotion, its various indicators, measurement, individual differences, dyadic interaction, and problems of inference.

REFERENCES

• 1: Ambadar, Z., Schooler, J. and Cohn, J. F. Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions. Psychological Science, 16. 403--410.
• 2: Boker, S. M., Xu, M., Rotondo, J. L. and King, K. Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods, 7 (1). 338--355.
• 3: Chow, S.-M., Ram, N., Boker, S. M., Fujita, F. and Clore, G. C. Emotion as a thermostat: representing emotion regulation using a damped oscillator model. Emotion, 5 (2). 208--225.
• 4: Cohn, J. F., Ambadar, Z. and Ekman, P. Observer-based measurement of facial expression with the Facial Action Coding System. in Coan, J. A. and Allen, J. B. eds. The handbook of emotion elicitation and assessment. Oxford University Press Series in Affective Science, Oxford University, New York, NY, In press.
• 5: Cohn, J. F. and Campbell, S. B. Influence of maternal depression on infant affect regulation. in Cicchetti, D. and Toth, S. L. eds. Developmental Perspectives on Depression, University of Rochester Press, Rochester, New York, 1992, 103--130.
• 6: Cohn, J. F. and Ekman, P. Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. in Harrigan, J. A., Rosenthal, R. and Scherer, K. eds. Handbook of nonverbal behavior research methods in the affective sciences, Oxford, New York, 2005, 9--64.
• 7: Cohn, J. F. and Elmore, M. Effects of contingent changes in mothers' affective expression on the organization of behavior in 3-month old infants. Infants Behavior and Development, 11. 493--505.
• 8: Cohn, J. F. and Kanade, T. Use of automated facial image analysis for measurement of emotion expression. in Coan, J. A. and Allen, J. B. eds. The handbook of emotion elicitation and assessment, Oxford, New York, NY, in press.
• 9: Cohn, J. F., Reed, L. I., Moriyama, T., Xiao, J., Schmidt, K. L. and Ambadar, Z., Multimodal coordination of facial action, head rotation, and eye motion. in Sixth IEEE International Conference on Automatic Face and Gesture Recognition, (Seoul, Korea, 2004), 645--650.
• 10: Cohn, J. F. and Schmidt, K. L. The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing, 2. 1--12.
• 11: Individual Differences in Facial Expression: Stability over Time, Relation to Self-Reported Emotion, and Ability to Inform Person Identification, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.491, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167045]
• 12: Cohn, J. F. and Tronick, E. Z. Mother-Infant face-to-face interaction: Influence is bidirectional and unrelated to periodic cycles in either partner's behavior. Developmental Psychology, 34 (3). 386--392.
• 13: Cohn, J. F. and Tronick, E. Z. Mother infant interaction: The sequence of dyadic states at three, six, and nine months. Developmental Psychology, 23. 68--77.
• 14: Darwin, C. The expression of the emotions in man and animals (3rd Edition). Oxford University, New York, New York, 1872/1998.
• 15: Davidson, R. J., Ekman, P., Saron, C. D., Senulis, J. A. and Friesen, W. V. Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology I. Journal of Personality and Social Psychology, 58 (2). 330--341.
• 16: Dimberg, U., Thunberg, M. and Grunedal, S. Facial reactions to emotional stimuli: Automatically controlled emotional responses. Cognition and Emotion, 16 (4). 449--471.
• 17: Edwards, K. The face of time: Temporal cues in facial expressions of emotion. Psychological Science, 9 (4). 270--276.
• 18: Eibl-Eibesfeldt, I. Human ethology. Aldine de Gruvier, NY, NY, 1989.
• 19: Ekman, P. Are there basic emotions? Psychological Review, 99 (3). 550--553.
• 20: Ekman, P. An argument for basic emotions. Cognition and Emotion, 6 (3/4). 169--200.
• 21: Ekman, P. Emotions revealed. Times, New York, NY, 2003.
• 22: Ekman, P. and Friesen, W.V. Facial action coding system. Consulting Psychologists Press, Palo Alto, CA, 1978.
• 23: Ekman, P., Friesen, W. V. and Hager, J. C. (eds.). Facial action coding system. Research Nexus, Network Research Information, Salt Lake City, UT, 2002.
• 24: Ekman, P., Friesen, W. V. and O'Sullivan, M. Smiles when lying. Journal of Personality and Social Psychology, 54 (3). 414--420.
• 25: Ekman, P. and Rosenberg, E. (eds.). What the face reveals. Oxford, New York, NY, 2005.
• 26: Fleiss, J. L. Statistical methods for rates and proportions. Wiley, New York, New York, 1981.
• 27: Frank, M. G. and Ekman, P. The ability to detect deceit generalizes across different types of high-stakes lies. Journal of Personality and Social Psychology, 72 (6). 1429--1439.
• 28: Fridlund, A. J., Sabini, J. P., Hedlund, L. E., Schaut, J. A., Shenker, J. J. and Knauer, M. J. Audience effects on solitary faces during imagery: Displaying to the people in your head. Journal of Nonverbal Behavior, 14 (2). 113--137.
• 29: Gottman, J., Levenson, R. and Woodin, E. Facial expressions during marital conflict Journal of Family Communication, 1 (1). 37--57.
• 30: Ibanez, L., Messinger, D., Ambadar, Z. and Cohn, J. F., Automated measurement of infant and mother interactive smiling. in American Psychological Society, (2006).
• 31: Jaffe, J., Beebe, B., Feldstein, S., Crown, C. L. and Jasnow, M. Rhythms of dialogue in early infancy. Monographs of the Society for Research in Child Development, 66 (2, Serial No. 264).
• 32: Kanade, T., Hu, C. and Cohn, J. F., Facial expression analysis. in IEEE International Workshop on Modeling and Analysis of Faces and Gestures, (Beijing, China, 2005).
• 33: Keltner, D. Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement and shame. Journal of Personality and Social Psychology, 68 (3). 441--454.
• 34: Keltner, D. and Ekman, P. Facial expression of emotion. in Lewis, M. and Haviland, J. M. eds. Handbooks of emotions, Guilford, New York, 2000, 236--249.
• 35: Keltner, D. and Haidt, J. Social functions of emotions at multiple levels of analysis. Cognition and Emotion, 13 (5). 505--522.
• 36: Krumhuber, E. and Kappas, A. Moving smiles: The role of dynamic components for the perception of the genuineness of smiles. Journal of Nonverbal Behavior, 29. 3--24.
• 37: Lazarus, R. S. Emotion and adaptation. Oxford, NY, 1991.
• 38: Levenson, R. W., Ekman, P. and Friesen, W. V. Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology, 27 (4). 363--384.
• 39: Malatesta, C. Z., Culver, C., Tesman, J. R. and Shephard, B. The development of emotion expression during the first two years of life. Monographs of the Society for Research in Child Development, 54 (219).
• 40: Malatesta, C. Z. and Haviland, J. M. Learning display rules: The socialization of emotion expression in infancy. Child Development, 53. 991--1003.
• 41: Messinger, D. S., Chow, S. M., Koterba, S., Hu, C., Haltigan, J. D., Wang, T. and Cohn, J. F. Continuously measured smile dynamics in infant-mother interaction, Miami, 2006.
• 42: Moore, G. A., Cohn, J. F. and Campbell, S. B. Mothers' affective behavior with infant siblings: Stability and change. Developmental Psychology., 33. 856--860.
• 43: Oster, H., Camras, L. A., Campos, J., Campos, R., Ujiee, T., Zhao-Lan, M. and Lei, W. The patterning of facial expressions in Chinese, Japanese, and American infants in fear- and anger- eliciting situations, Poster presented at the International Conference on Infant Studies, Providence, RI, 1996.
• 44: Oster, H., Hegley, D. and Nagel, L. Adult judgments and fine-grained analysis of infant facial expressions: Testing the validity of a priori coding formulas. Developmental Psychology, 28 (6). 1115--1131.
• 45: Pantic, M. and Patras, I. Dynamics of facial expressions: Recognition of facial actions and their temporal segments from profile image sequences. IEEE Tansactions on Systems, Man, and Cybernetics, Part B, 36 (2). 443--449.
• 46: Maja Pantic , Leon J. M. Rothkrantz, Automatic Analysis of Facial Expressions: The State of the Art, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1424-1445, December 2000 [doi>10.1109/34.895976]
• 47: Pantic, M. and Rothkrantz, M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91. 1371--1390.
• 48: Maja Pantic , Nicu Sebe , Jeffrey F. Cohn , Thomas Huang, Affective multimodal human-computer interaction, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101299]
• 49: Rosenthal, R. Conducting judgment studies. in Harrigan, J. A., Rosenthal, R. and Scherer, K. R. eds. Handbook of nonverbal behavior research methods in the affective sciences, Oxford, NY, 2005, 199--236.
• 50: Schmidt, K. L. and Cohn, J. F. Human facial expressions as adaptations: Evolutionary perspectives in facial expression research. Yearbook of Physical Anthropology, 116. 8--24.
• 51: Tian, Y., Cohn, J. F. and Kanade, T. Facial expression analysis. in Li, S. Z. and Jain, A. K. eds. Handbook of face recognition, Springer, New York, New York, 2005, 247--276.
• 52: Michel F. Valstar , Maja Pantic , Zara Ambadar , Jeffrey F. Cohn, Spontaneous vs. posed facial behavior: automatic analysis of brow actions, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181031]
• 53: Watson, D. and Tellegen, A. Toward a consensual structure of mood. Psychological Bulletin, 98 (2). 219--235.
• 54: Zlochower, A. J. and Cohn, J. F. Vocal timing in face-to-face interaction of clinically depressed and nondepressed mothers and their 4-month-old infants. Infant Behavior and Development, 19. 373--376.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Psychology

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Modeling and recovery of physical attributes; Motion; Video analysis

General Terms:
Design, Human Factors, Measurement, Reliability

Keywords:
automatic facial image analysis, emotion, facial expression, human-computer interaction, temporal dynamics