Faces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain

ABSTRACT

We present initial results from the application of an automated facial expression recognition system to spontaneous facial expressions of pain. In this study, 26 participants were videotaped under three experimental conditions: baseline, posed pain, and real pain. In the real pain condition, subjects experienced cold pressor pain by submerging their arm in ice water. Our goal was to automatically determine which experimental condition was shown in a 60 second clip from a previously unseen subject. We chose a machine learning approach, previously used successfully to categorize basic emotional facial expressions in posed datasets as well as to detect individual facial actions of the Facial Action Coding System (FACS) (Littlewort et al, 2006; Bartlett et al., 2006). For this study, we trained 20 Action Unit (AU) classifiers on over 5000 images selected from a combination of posed and spontaneous facial expressions. The output of the system was a real valued number indicating the distance to the separating hyperplane for each classifier. Applying this system to the pain video data produced a 20 channel output stream, consisting of one real value for each learned AU, for each frame of the video. This data was passed to a second layer of classifiers to predict the difference between baseline and pained faces, and the difference between expressions of real pain and fake pain. Naíve human subjects tested on the same videos were at chance for differentiating faked from real pain, obtaining only 52% accuracy. The automated system was successfully able to differentiate faked from real pain. In an analysis of 26 subjects, the system obtained 72% correct for subject independent discrimination of real versus fake pain on a 2-alternative forced choice. Moreover, the most discriminative facial action in the automated system output was AU 4 (brow lower), which all was consistent with findings using human expert FACS codes.

REFERENCES

• 1: Bartlett M. S., Littlewort G. C., Frank M. G., Lainscsek C., Fasel I., and Movellan J. R., "Automatic recognition of facial actions in spontaneous expressions.," Journal of Multimedia., 1(6) p. 22--35.
• 2: Cohn, J. F. & Schmidt, K. L. (2004). The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing, Vol. 2, No. 2, pp. 121--132.
• 3: Craig K. D, Hyde S., Patrick C. J. (1991). Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46:161--172.
• 4: Craig K. D, Patrick C. J. (1985). Facial expression during induced pain. J Pers Soc Psychol. 48(4):1080--91.
• 5: Gianluca Donato , Marian Stewart Bartlett , Joseph C. Hager , Paul Ekman , Terrence J. Sejnowski, Classifying Facial Actions, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.21 n.10, p.974-989, October 1999 [doi>10.1109/34.799905]
• 6: Ekman P. and Friesen, W. Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press, Palo Alto, CA, 1978.
• 7: Ekman, P. (2001). Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York, USA.
• 8: Ekman, P. & Rosenberg, E. L., (Eds.), (2005). What the face reveals: Basic and applied studies of spontaneous expression using the FACS, Oxford University Press, Oxford, UK.
• 9: Ian Fasel , Bret Fortenberry , Javier Movellan, A generative framework for real time object detection and classification, Computer Vision and Image Understanding, v.98 n.1, p.182-210, April 2005 [doi>10.1016/j.cviu.2004.07.014]
• 10: Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (1999). Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain. 15(4):244--74.
• 11: Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (2006). Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3):214--34.
• 12: Frank M. G., Ekman P., Friesen W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1):83--93.
• 13: Grossman, S., Shielder, V., Swedeen, K., Mucenski, J. (1991). Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), p. 53--57.
• 14: Hadjistavropoulos H. D., Craig K. D., Hadjistavropoulos T., Poole GD. (1996). Subjective judgments of deception in pain expression: accuracy and errors. Pain. 65(2-3):251--8.
• 15: Hill M. L., Craig K. D. (2002) Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain. 98(1-2):135--44.
• 16: Takeo Kanade , Yingli Tian , Jeffrey F. Cohn, Comprehensive Database for Facial Expression Analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.46, March 26-30, 2000
• 17: Larochette A. C., Chambers C. T., Craig K. D. (2006). Genuine, suppressed and faked facial expressions of pain in children. Pain. 2006 Dec 15;126(1-3):64--71.
• 18: Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J. & Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing, Vol. 24, No. 6, pp. 615--625.
• 19: Morecraft R. J, Louie J. L., Herrick J. L., Stilwell-Morecraft KS. (2001). Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1):176--208.
• 20: Maja Pantic , Alex Pentland , Anton Nijholt , Thomas Huang, Human computing and machine understanding of human behavior: a survey, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181044]
• 21: Pantic, M. F. Valstar, R. Rademaker and L. Maat, "Web-based Database for Facial Expression Analysis", Proc. IEEE Int'l Conf. Multmedia and Expo (ICME'05), Amsterdam, The Netherlands, July 2005.
• 22: Prkachin K. M. (1992). The consistency of facial expressions of pain: a comparison across modalities. Pain. 51(3):297--306.
• 23: Prkachin K. M., Schultz I., Berkowitz J., Hughes E., Hunt D. Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5):595--607.
• 24: Rinn W. E. The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95:52--77.
• 25: Schmand B., Lindeboom J., Schagen S., Heijt R., Koene T., Hamburger H. L. Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry. 64(3):339--43.
• 26: Schmidt K. L., Cohn J. F., Tian Y. (2003). Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1):49--66.
• 27: H. Schneiderman , T. Kanade, Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.45, June 23-25, 1998
• 28: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 29: Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., and Movellan, J.R. (2007).allMachine learning systems for detecting driver drowsiness. Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Medical information systems

Additional Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Psychology

General Terms:
Design, Experimentation, Human Factors, Measurement, Performance

Keywords:
FACS, computer vision, deception, facial action coding system, facial expression recognition, machine learning, pain, spontaneous behavior