Spontaneous vs. posed facial behavior: automatic analysis of brow actions

ABSTRACT

Past research on automatic facial expression analysis has focused mostly on the recognition of prototypic expressions of discrete emotions rather than on the analysis of dynamic changes over time, although the importance of temporal dynamics of facial expressions for interpretation of the observed facial behavior has been acknowledged for over 20 years. For instance, it has been shown that the temporal dynamics of spontaneous and volitional smiles are fundamentally different from each other. In this work, we argue that the same holds for the temporal dynamics of brow actions and show that velocity, duration, and order of occurrence of brow actions are highly relevant parameters for distinguishing posed from spontaneous brow actions. The proposed system for discrimination between volitional and spontaneous brow actions is based on automatic detection of Action Units (AUs) and their temporal segments (onset, apex, offset) produced by movements of the eyebrows. For each temporal segment of an activated AU, we compute a number of mid-level feature parameters including the maximal intensity, duration, and order of occurrence. We use Gentle Boost to select the most important of these parameters. The selected parameters are used further to train Relevance Vector Machines to determine per temporal segment of an activated AU whether the action was displayed spontaneously or volitionally. Finally, a probabilistic decision function determines the class (spontaneous or posed) for the entire brow action. When tested on 189 samples taken from three different sets of spontaneous and volitional facial data, we attain a 90.7% correct recognition rate.

REFERENCES

• 1: Ambadar, Z., Schooler, J. and Cohn, J.F. Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions, Psychological Science, 16, 5 (May 2005), 403--410.
• 2: Ambady, N. and Rosenthal, R. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin, 111, 2 (Feb. 1992), 256--274.
• 3: Bartlett, M. S., Hager, J. C., Ekman, P. and Sejnowski, T. J. Measuring facial expressions by computer image analysis. Psychophysiology, 36, 2 (Mar. 1999), 253--263.
• 4: Marian Stewart Bartlett , Gwen Littlewort , Mark Frank , Claudia Lainscsek1 , Ian Fasel , Javier Movellan, Fully Automatic Facial Action Recognition in Spontaneous Behavior, Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), p.223-230, April 10-12, 2006 [doi>10.1109/FGR.2006.55]
• 5: Bartlett, M. S., Littlewort, G., Lainscsek, C., Fasel, I. and Movellan, J. Machine learning methods for fully automatic recognition of facial expressions and actions. In Proc. Int'l Conf. Systems, Man and Cybernetics, 592--597, 2004.
• 6: Bassili, J. N. Facial motion in the perception of faces and of emotional expression", J. Experimental Psychology, 4, 3 (Aug. 1978), 373--379.
• 7: Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J. and Moriyama, T. Automatic analysis and recognition of brow actions in spontaneous facial behavior. In Proc. Int'l Conf. Systems, Man & Cybernetics, 1, 610--616, 2004.
• 8: Cohn, J. F. and Schmidt, K. L. The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing, 2, 2 (June 2004), 121--132.
• 9: Douglas W. Cunningham , Mario Kleiner , Heirich H. Bülthoff , Christian Wallraven, The components of conversational facial expressions, Proceedings of the 1st Symposium on Applied perception in graphics and visualization, August 07-08, 2004, Los Angeles, California [doi>10.1145/1012551.1012578]
• 10: Duchenne de Bologne, G. B. Mechanisme de la Physionomie Humaine. Jules Renouard Libraire, Paris, France, 1862. (Translation: The Mechanism of Human Facial Expression. Cambridge University Press, New York, 1990).
• 11: Ekman, P. Darwin, deception, and facial expression. Annals New York Academy of sciences, 1000, (Dec. 2003), 205--221.
• 12: Ekman, P. and Friesen, W. F. Felt, false and miserable smiles. J. Nonverbal Behavior, 6, 4 (June 1982), 238--252.
• 13: Ekman, P., Friesen, W. V. and Hager, J. C. Facial Action Coding System. A Human Face, Salt Lake City, 2002.
• 14: Ekman, P., Hager, J. C. and Friesen, W. V. The symmetry of emotional and deliberate facial actions. Psychophysiology, 18, 2 (Mar. 1981), 101--106.
• 15: Ekman, P. and Rosenberg, E. L. Eds. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System. Oxford University Press, Oxford, UK, 2005.
• 16: Rana El Kaliouby , Peter Robinson, Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10, p.154, June 27-July 02, 2004
• 17: Beat Fasel , Florent Monay , Daniel Gatica-Perez, Latent semantic analysis of facial action codes for automatic facial expression recognition, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026742]
• 18: Flecha-Garcia, M. L. Facial gestures and communication: what induces raising eyebrow movements in Map Task dialogues. Proc. Theoretical & Applied Linguistics -- Postgraduate Conf., 2001.
• 19: Fogel, A., Hsu, H. C., Shapiro, A. F., Nelson-Goens, G. C. and Secrist, C. Effects of normal and perturbed social play on the duration and amplitude of different types of infant smiles. Developmental Psychology, 42, 3 (May 2006), 159--173.
• 20: Friedman, J., Hastie, T. and Tibshirani, R. Additive logistic regression: a statistical view of boosting. The Annals of Statistics, 28, 2 (Apr. 2000), 337--374.
• 21: Lisa Gralewski , Neill Campbell , Ian Penton-Voak, Using a Tensor Framework for the Analysis of Facial Dynamics, Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), p.217-222, April 10-12, 2006 [doi>10.1109/FGR.2006.108]
• 22: Gu, H. and Ji, Q. An automated face reader for fatigue detection. In Proc. Int'l Conf. Face & Gesture Recognition, 111--116, 2004.
• 23: Hess, U. and Kleck, R. E. Differentiating emotion elicited and deliberate emotional facial expressions. European J. of Social Psychology, 20, 5 (Sep./Oct. 1990), 369--385.
• 24: Takeo Kanade , Yingli Tian , Jeffrey F. Cohn, Comprehensive Database for Facial Expression Analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.46, March 26-30, 2000
• 25: Kendon, A. A description of some human greetings. In Comparative Ecology and Behavior of Primates, Michael, R. and Crook, J., Eds. Academic Press, New York, USA, 1973, 591--668.
• 26: Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J. and Movellan, J. Dynamics of facial expression extracted automatically from video. Int'l J. Image & Vision Computing, 24, 6 (June 2006), 615--625.
• 27: Lucas, B. and Kanade, T. An interactive image registration technique with an application in stereo vision. In Proc. Int'l Joint Conf. on Artificial Intelligence, 674--679, 1981.
• 28: Messinger, D. S., Fogel, A. and Dickson, K. L. A dynamic systems approach to infant facial action. The psychology of facial expression. Russell, J. A. and Fernandez-Dols, J. M., Eds. Cambridge University Press, Cambridge, UK, 1997, 205--226.
• 29: Pantic, M. and Patras, I. Dynamics of facial expressions -- recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Systems, Man, and Cybernetics, Part B, 36, 2 (Apr. 2006), 433--449.
• 30: Maja Pantic , Alex Pentland , Anton Nijholt , Thomas Huang, Human computing and machine understanding of human behavior: a survey, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181044]
• 31: Pantic, M. and Rothkrantz, L. J. M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91, 9 (Sep. 2003), 1370--1390.
• 32: Pantic, M. and Rothkrantz, L. J. M. Case-based reasoning for user-profiled recognition of emotions from face images. In Proc. Int'l Conf. Multimedia and Expo, 2004, 391--394.
• 33: Pantic, M., Valstar, M. F., Rademaker, R. and Maat, L. Web-based database for facial expression analysis. In Proc. Int'l Conf. Multimedia and Expo, 2005, 317--321. (www.mmifacedb.com)
• 34: Patras, I. and Pantic, M. Particle filtering with factorized likelihoods for tracking facial features. In Proc. Int'l Conf. Automatic Face & Gesture Recognition, 97--102, 2004.
• 35: Patras, I. and Pantic, M. Tracking deformable motion. In Proc. Int'l Conf. Systems, Man and Cybernetics, 1066--1071, 2005.
• 36: Rinn, W. E. The neuropsychology of facial expressions: A review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, 95, 1 (Jan. 1984), 52--77.
• 37: Rosenberg, E. L., Ekman, P. and Blumenthal, J. A. Facial expression and the affective component of cynical hostility in male coronary heart disease patients. J. on Health Psychology, 17, 4 (July 1998), 376--380.
• 38: Schmidt, K. L., Ambadar, Z., Cohn, J. F. and Reed, I. Movement differences between deliberate and spontaneous facial expressions: Zygomaticus Major action in smiling. J. Nonverbal Behavior, 30, 1 (Mar. 2006), 37--52.
• 39: Tian, Y. L., Kanade, T. and Cohn, J. F. Facial Expression Analysis. In Handbook of Face Recognition. Li, S. Z. and Jain, A. K., Eds. Springer, New York, 2005.
• 40: Tipping, M. E. The relevance vector machines. In Proc. Int'l Conf. Adv. Neural Inf. Processing Systems, 332--338, 2000.
• 41: Michel Valstar , Maja Pantic, Fully Automatic Facial Action Unit Detection and Temporal Analysis, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, p.149, June 17-22, 2006 [doi>10.1109/CVPRW.2006.85]
• 42: Veldhuizen, I. J. T., Gaillard, A. W. K. and De Vries, J. The influence of mental fatigue on facial EMG activity during a simulated workday. Biological Psychology, 63, 1 (Apr. 2003), 59--78.
• 43: Vukadinovic, D. and Pantic, M. Fully automatic facial feature point detection using Gabor feature based boosted classifiers. In Proc. Int'l Conf. Systems Man and Cybernetics, 1692--1698, 2005.
• 44: Williams, A. C. de C. Facial expression of pain: An evolutionary account. Behavioral and Brain Sciences, 25, 4 (Aug. 2002), 439--488.
• 45: Xiao, J., Moriyama, T., Kanade, T. and Cohn, J. F. Robust full-motion recovery of head by dynamic templates and re-registration techniques. Int'l J. Imaging Systems and Technology, 13, 1 (Sep. 2003), 85--94.
• 46: Yongmian Zhang , Qiang Ji, Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.5, p.699-714, May 2005 [doi>10.1109/TPAMI.2005.93]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Modeling and recovery of physical attributes

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
automatic facial expression analysis, temporal dynamics