ACM Home Page
Please provide us with feedback. Feedback
Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature
Full text PdfPdf (2.58 MB)
Source
ACM Transactions on Applied Perception (TAP) archive
Volume 6 ,  Issue 1  (February 2009) table of contents
Article No. 6  
Year of Publication: 2009
ISSN:1544-3558
Authors
Masood Mehmood Khan  Curtin University of Technology, Perth, Western Australia
Robert D. Ward  University of Huddersfield, Queensgate, England
Michael Ingleby  University of Huddersfield, Queensgate, England
Publisher
ACM  New York, NY, USA
Bibliometrics
Downloads (6 Weeks): 43,   Downloads (12 Months): 224,   Citation Count: 0
Additional Information:

abstract   references   index terms   collaborative colleagues  

Tools and Actions: Request Permissions Request Permissions    Review this Article  
DOI Bookmark: Use this link to bookmark this Article: http://doi.acm.org/10.1145/1462055.1462061
What is a DOI?

ABSTRACT

Earlier researchers were able to extract the transient facial thermal features from thermal infrared images (TIRIs) to make binary distinctions between the expressions of affective states. However, effective human-computer interaction would require machines to distinguish between the subtle facial expressions of affective states. This work, for the first time, attempts to use the transient facial thermal features for recognizing a much wider range of facial expressions. A database of 324 time-sequential, visible-spectrum, and thermal facial images was developed representing different facial expressions from 23 participants in different situations. A novel facial thermal feature extraction, selection, and classification approach was developed and invoked on various Gaussian mixture models constructed using: neutral and pretended happy and sad faces, faces with multiple positive and negative facial expressions, faces with neutral and six (pretended) basic facial expressions, and faces with evoked happiness, sadness, disgust, and anger. This work demonstrates that (1) infrared imaging can be used to observe the affective-state-specific facial thermal variations, (2) pixel-grey level analysis of TIRIs can help localise significant facial thermal feature points along the major facial muscles, and (3) cluster-analytic classification of transient thermal features can help distinguish between the facial expressions of affective states in an optimized eigenspace of input thermal feature vectors. The observed classification results exhibited influence of a Gaussian mixture model's structure on classifier-performance. The work also unveiled some pertinent aspects of future research on the use of facial thermal features in automated facial expression classification and affect recognition.


REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

 
1
Abidi, B., Huq, S., and Abidi, M. 2004. Fusion of visual, thermal, and range as a solution to illumination and pose restrictions in face recognition. In Proceedings of the 2004 IEEE International Carnahan Conference on Security Technology. IEEE, Los Alamitos, CA, 325--330.
 
2
 
3
Allanson, J. and Fairclough, S. H. 2004. A research agenda for physiological computing. Interact. Comput. 16, 857--878.
 
4
Ang, L. B. P., Belen, E. F., Bernardo, R. A., Boongaling, JR., E. R., Briones, G. HH., Asthana, H. S., and Mandal, M. K. 1997. Hemiregional variations in facial expression of emotions. British J. Psych. 88, 519--525.
 
5
Bartlett, M. S., Hager, J. C., Ekman P., and Sejnowski, T. J. 1999. Measuring facial expressions by computer image analysis. J. Psychophysiol. 36, 253--263.
 
6
Belmont report. 1979. Ethical principles and guidelines for the protection of human subjects of research. United States Government, National Institute of Health, Washington, DC. http://ohsr.od.nih.gov/guidelines/belmont.html.
 
7
 
8
 
9
 
10
Bradley, M. M., Sabatinelli, D., Lang, P. J., Fitzsimmons, J. R., King, W., and Desai, P. 2003. Activation of the visual cortex in motivated attention. Behav. Neuroscience 117, 369--380.
 
11
Breitsprecher, L., Fanghånel, A., Noe, A., Lockett, E., and Raab, U. 2002. The functional anatomy of the muscles of facial expression in humans with and without Cleft Lip and Palate: A contribution to refine muscle reconstruction in primary cheilo and rhinoplasties in patients with uni and bilateral complete CLP. Annals Anatomy 184, 27--34.
12
 
13
Calder, A. J., Burton, A. M., Miller, P., Young, A. W., and Akamatsu, S. 2001. A principal component analysis of facial expressions. Vision Resear. 41, 1179--1208.
 
14
CANTRONIC SYSTEMS INC. 2001. IR 860 User Manual. Cantronic Systems Inc. Coquitlam, Canada.
 
15
 
16
Christie, I. C. and Friedman, B. H. 2004. Autonomic specificity of discrete emotion and dimensions of affective space: A multivariate approach. Int. J. Psychophysiol. 51, 143--153.
 
17
 
18
Collet, C., Vernet-Maury, E., Delhomme, G., and Dittmar, A. 1997. Autonomic nervous system response patterns specificity to basic emotions. J. Autonomic Nervous Syst. 63, 45--57.
 
19
Crissey, J. T. and Parish, L. C. 2004. The red face: Historical Considerations. Clinics Dermatology 11, 2, 197--201.
 
20
Critchley, D., Daly, E. M., Bullmore, E. T., Williams, S. C. R., Amelsvoort, T. V., Robertson, D. M., Rowe, A., Phillips, M., McAlonan, G., Howllin, P., and Murphy, D. G. M. 2000. The functional neuroanantomy of social behavior: Changes in cerebral blood flow when people with autistic disorder process facial expressions. Brain 123, 203--212.
 
21
DeSilva, L. C., Miyasato, T., and Nakatsu, R. 1997. Facial emotion recognition using multimodal information. In Proceedings of the IEEE Information, Communication and Signal Processing Conference. IEEE, Los Alamitos, CA, 397--401.
 
22
dhew. 1979. Department of Health, Education and Welfare. Publication No. (OS) 78-0014. US Government Printing Office, Washington, DC.
 
23
Dimberg, U. 1990a. Facial electromyography and emotional reactions. Psychophysiology 27, 3, 481--494.
 
24
Dimberg, U. 1990b. Facial reactions to auditory stimuli: Sex differences. Scandinavian J. Psych. 31, 3, 228--233.
 
25
Dror, I. E., Péron, A. E., Hind, S. L., and Charlton, D. 2005. When emotions get better of us: The effect of contextual top-down processing on matching Fingertips. Appl. Cognitive Psych. 19, 799--809.
 
26
Dubuisson, S., Davoine, F., and Masson, M. 2002. A solution for facial expression representation and recognition. Signal Process.: Image Comm. 17, 657--673.
 
27
 
28
 
29
Ekman, P. 1982. Emotion in the Human Face. Cambridge University Press, UK.
 
30
Ekman, P., Davidson, R. J., and Friesen, W. V. 2000. Duchenne's smiles: Emotional expression and their brain physiology II. J. Person. Social Psychol. 58, 342--353.
 
31
Ekman, P., Levenson, R. W., and Friesen, W. V. 1983. Autonomic nervous system activity distinguishes among emotions. Science 221, 1208--1210.
 
32
 
33
Eveland, C. K., Socolinsky, D. A., and Wolf, L. B. 2003. Tracking human faces in infrared video. Image Vision Comput. 21, 7, 579--590.
 
34
Everitt, B. S. and Dunn, G. 1991. Applied Multivariate Data Analysis. John Wiley and Sons, London.
 
35
Fujimasa, I. 1998. Pathophysiological expression and analysis of infrared thermal images. IEEE Engin. Medicine Biology 17, 34--42.
 
36
Fujimasa, I., Chinzei, T., and Saito, I. 2000. Converting far infrared image information to other physiological data. IEEE Engin. Medicine and Biology 19, 3, 71--76.
 
37
Garbey, M., Merla, A., and Pavlidia, I. 2004. Estimation f blood flow speed and vessel location from thermal video. In Proceedings of CVRP 2004-IEEE 2004 Conference on Computer Vision and Pattern Recognition 1, Washington DC, 1356--1363.
 
38
 
39
Healey, J. A. and Picard, R. W. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transport. Syst. 6, 156--166.
 
40
Herry, C. L. and Frize, M. 2002. Digital processing techniques for the assessment of pain with infrared thermal imaging. In Proceedings of the IEEE International Conference on Engineering in Medicine and Biology (EMBS'02). IEEE, Los Alamitos, CA, 1157--1158.
 
41
Hess, U., Kappas, A., McHugo, G. J., Lanzetta, J. T., and Kleck, R. E. 1992. The facilitative effect of facial expression on self-generation of emotion. Psychophysiology 12, 3, 251--265.
 
42
Huang, C. and Huang, Y. 1997. Facial expression recognition using model-based feature extraction and action parameters classification. J. Visual Comm. Image Represen. 8, 3, 278--290.
 
43
IAPS. 1997. International affective picture system technical manual and affective ratings. http://www.unifesp.br/adap/instructions.pdf.
 
44
Iwase, M., Ouchi, Y., Okada, H., Yokohama, C., Nobezawa, S., Yoshikawa, E., Tsukada, H., Takeda, M., Yamagguti, K., Kuratsune, H., Shimizu, A., and Watanabe, Y. 2002. Neural substrates of human facial expression of pleasant emotion induced by comic films: a PET study. Neuroimage 17, 758--768.
 
45
Jiang, G. and Le, K. 2007. Character analysis of facial expression image. In Proceedings of the IEEE/ICME International Conference on Complex Medical Engineering (CME'07). 824--827.
 
46
Jolliffe, I. T. 2002. Principal Component Analysis. Springer-Verlag, New York.
 
47
Jones, B. F. and Plassmann, P. 2002. Digital infrared thermal imaging of human skin. IEEE Engin. Medicine Biology 21, 41--48.
 
48
Jones, B. F. 1998. A reappraisal of use of infrared thermal image analysis in medicine. IEEE Trans. Medical Imag. 17, 1019--1027.
 
49
Jones, C. H., Ring, E. F. J., and Clark, R. P. 1988. Medical Thermography. In Applications of Thermal Imaging, Burnay, S. G., Williams, T. L. and Jones, C. H. N., Eds. Adam Hilger, Bristol, 156--187.
 
50
 
51
 
52
Kall, R. 1990. Emotional self regulation and facial expression muscle measurement and training. In Clinical EMG for Surface Recording, vol. 2, J. R. Cram and J. V. Basmalian, Eds. Clinical Resources, Nevada City, CA.
 
53
Khan, M. M. 2008. Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features. Ph.D. Thesis, Department of Computing and Engineering, University of Huddersfield, Huddersfield, UK.
54
 
55
Khan, M. M., Ward, R. D., and Ingleby, M. 2004. Automated classification and recognition of facial expressions using infrared thermal imaging. In Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems. IEEE, Los Alamitos, CA, 202--206.
 
56
Khan, M. M., Ward, R. D., and Ingleby, M. 2005. Distinguishing facial expressions by thermal imaging using facial thermal feature points. In Proceedings of the 19th British HCI Group Annual Conference (HCI'05). Edinburgh, UK.
 
57
Kim, H. K., Bang, S. W., and Kim, S. R. 2004. Emotion recognition system using short-term monitoring of physiological signals. Medical Biological Engin. Comput. 42, 419--427.
 
58
Kinnear, P. R. and Gray, C. D. 2000. SPSS for Windows Made Simple. Psychology Press Ltd., East Sussex, UK.
 
59
Kobayashi, H. and Tagami, H. 2004. Distinct locational differences observable in biophysical functions of the facial skin: with special emphasis on the poor functional properties of the stratum corneum of the perioral region. Int. J. Cosmetic Science 26, 91--101.
 
60
 
61
Kunzmann, U. and Gruhn, D. 2005. Age differences in emotional reactivity: the sample case of sadness. Psych. Aging 20, 47--59.
 
62
Kurse, P. W. 2001. Uncooled Thermal Imaging: Analysis, Systems and Applications. SPIE Press,
 
63
 
64
Lyons, M., Budynek, J., and Akamatsu, S. 1999. Classifying images of facial expression using a Gabor wavelet representation. In Proceedings of 2nd International Conference on Cognitive Science, 113--118.
 
65
Matsuzaki, H. and Mizote, M. 1996. Measurement of facial temperature fluctuations by thermal image analysis. Progress Biophysics Molec. Biology 65, 185--186.
 
66
Mcfarland, R. A. and Kadish, R. 1991. Sex differences in finger temperature response to music. Int. J. Psychophysiol. 11, 295--298.
 
67
Mcgimpsey, J. G., Vaidya, A., Biagioni, P. A., and Lamey, P.-J. 2000. Role of thermography is the assessment of infraorbital nerve injury after malar fractures. British J. Oral Maxillofacial Surgery 38, 581--584.
 
68
Mclachlan, G. J. 2004. Discriminant Analysis and Statistical Pattern Recognition. Wiley, New Jersey.
 
69
Murphy, S. T. and Zajonc, R. B. 1993. Affect, cognition and awareness: affective priming with optimal and suboptimal stimulus exposures. J. Personality Social Psych. 64, 723--739.
 
70
Naemura, A., Tsuda, K., and Suzuki, N. 1993. Effects of loud noise on nasal skin temperature (Originally published in Japanese, translation available). Shingrigaku Kenkyu 64, 51--54.
 
71
Nakayama, K., Goto, S., Kuraoka, K., and Nakamura, K. 2005. Decrease in nasal temperature of rhesus monkeys (Macaca mulatta) in negative emotional state. J. Physiol. Behav. 84, 783--790.
 
72
Niedenthal, P. M., Halberstadt, J. B., Margolin, J., and Innes-Ker, A. H. 2000. Emotional state and the detection of change in facial expression of emotion. European J. Social Psychol. 30, 211--222.
 
73
Ogasawara, T., Kitagawa, Y., Ogawa, T., Yamada, T., Kawamura, Y., and Sano, K. 2001. MR Imaging and Thermography of facial angioedema: A case report. Oral Surgery, Oral Medicine, Oral Pathology 92, 473--476.
 
74
Olivier, B., Zethof, T., Pattij, T., van Boogaert, M., van Oorshot, R., Leahy, C., Oosting, R., Bouwknecht, A., Veening, J., van der, Gugten, and Groenink, L. 2003. Stress-induced hypothermia and anxiety: Pharmacological validation. European J. Pharmacol. 28, 117--132.
 
75
Otsuka, K., Okada, S., Hassan, M., and Togawa, T. 2002. Imaging of skin thermal properties with estimation of ambient radiation. IEEE Engin. Medicine Biology 21, 4, 49--55.
 
76
Palomba, D., Sarlo, M., Angrilli, A., Mini, A., and Stegagno, L. 2000. Cardiac responses associated with affective processing of unpleasant film stimuli. Inter. J. Psychophysiol. 36, 45--57.
 
77
 
78
Pavlidis, I. and Levine, J. 2002. Thermal image analysis for polygraph testing. IEEE Engin. Medicine Biology 21, 56--64.
 
79
Pavlidis, I. 2004. Lie detection using thermal imaging. In Proceedings of SPIE - Thermosense XXVI, Conference of the International Society for Optical Engineering. 270--279.
 
80
Pessa, J., Zadoo, V., Gerza, P., Adrian, E. J., Dewitt, A., and Garza, J. 1998. Double or bifid zygomatcius major muscle: Anatomy, incidence and clinical correlation. J. Clinical Anatomy 11, 310--313.
 
81
Pham, T. H., Philippot, P., and Rime, B. 2000. Subjective and autonomic responses to emotion induction in psychopaths. Encaphale 26, 45--51.
 
82
Phillips, T. J. 2002. High performance thermal imaging technology. Advanced Semiconductor Magazine 15, 32--36.
 
83
 
84
Pizzagalli, D., Koenig, T., Regard, M., and Lehmann, D. 1998. Faces and emotion: brain electric field sources during covert emotional processing. Neuropsychologia 36, 323--332.
 
85
Pollina, D. A., Dollins, A. B., Senter, S. M., Brown, T. E., Pavlidis, I., Levine, J. A., and Ryan, A. H. 2006. Facial skin surface temperature changes during a concealed information test. Annals Biomed. Engin. 34, 7, 1182--1189.
 
86
Posamentier, M. T. and Abdi, H. 2003. Processing faces and facial expressions. Neuropsychol. Rev. 13, 113--143.
 
87
Prokoski, F. J. and Iedel, R. 1999. Infrared identification of faces and body parts. In Biometrics: Personal Identification in Networked Society. Jain, A. K., Bolle, R. M. and Pankanti, S., Eds. Kulwer Academic Press, Boston, MA. 191--212.
88
 
89
Rimm-Kaufman, S. E. and Kagan, J. 1996. The psychological significance of changes in skin temperature. Motivation Emotion 20, 63--78.
 
90
Root, A. A. and Stephens, J. A. 2003. Organization of the central control of muscles of facial expression in man. J. Physiol. 549, 289--298.
 
91
Sarlo, M., Buodo, G., Poli, S., and Palomba, D. 2005. Changes in EEG alpha power to different disgust elicitors: The specificity of mutilations. Neuroscience Lett. 382, 291--296.
 
92
Schwartz, G. E., Fair, P. L., Salt, P., Mandel, M. R., and Klerman, G. L. 1976. Facial muscle patterning to affective imagery in depressed and non-depressed subjects. Science, New Series 192, 489--491.
 
93
Schwarz, S., Hofmann, M. H., Gutzen, C., Schlax, S., and Emde, von der G. 2002. VIEWER: A program for visualizing, recording, and analyzing animal behavior. Comput. Meth. Programs Biomedicine 67, 55--66.
 
94
 
95
Sinha, R. and Parson, O. 1996. Multivariate response patterning of fear and anger. Cognition Emotion 10, 2, 173--198.
 
96
 
97
Sugimoto, Y., Yoshitomi, Y., and Tomita, S. 2000. A method of detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions. Robot. Autonom. Syst. 31, 147--160.
 
98
 
99
Toivanen, J., Våyrynen, E., and Seppånen, T. 2004. Automatic discrimination of emotions in spoken Finnish. Lang. Speech 47, 4, 383--412.
 
100
Vianna, D. M. and Carrive, P. 2005. Changes in cutaneous and body temperature during and after conditioned fear to context in the rat. European J. Neuroscince 21, 2505--2512.
 
101
Vrana, S. R. 1993. The psychophysiology of disgust: differentiating negative emotional contexts with facial EMG. Psychophysiol. 30 3, 279--286.
 
102
Vrana, S. R. and Gross, D. 2004. Reactions to facial expressions: effects of social context and speech anxiety on response to neutral, anger and joy expressions. Biological Psych. 66, 1, 63--78.
 
103
Webb, A. R. 2002. Statistical Pattern Recognition. John Wiley, London.
 
104
Whiteside, S. 1998. In Proceedings of the International Conference on Spoken Language Processing (ICSLP'98). 699--703.
 
105
Wild, B., Erb, M., and Bartels, M. 2001. Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: Quality, quantity, time course and gender differences. Psychiatry Resear. 102, 109--124.
 
106
Wolf, K., Raedler, T., Henke, K., Kiefer, F., Mass, R., Quante, M., and Wiedemann, K. 2005. The face of pain- a pilot study to validate the measurement of facial pain expression with improved electromyogram method. Pain Resear. Manag. 10, 1, 9--14.
 
107
Wright, P., He, G., and Shapira, N. A., Goodman, W. K., and Liu, Y. 2004. Disgust and the insula: fMRI responses to pictures of mutilation and contamination. NeuroReport 15, 2347--2351.
 
108
Yoshitomi, Y., Kim, S. I., Kawano, T., and Kitazoe, T. 2000. Effects of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In Proceedings of the IEEE International Workshop on Robotics and Human Interactive Communication. IEEE, Los Alamitos, CA, 178--183.
 
109
Zajonc, R. B. 1985. Emotion and facial efference: a theory reclaimed. Science 228, 15--21.

Collaborative Colleagues:
Masood Mehmood Khan: colleagues
Robert D. Ward: colleagues
Michael Ingleby: colleagues