Fusion of audio and visual cues for laughter detection

ABSTRACT

Past research on automatic laughter detection has focused mainly on audio-based detection. Here we present an audio-visual approach to distinguishing laughter from speech and we show that integrating the information from audio and video channels leads to improved performance over single-modal approaches. Each channel consists of 2 streams (cues), facial expressions and head movements for video and spectral and prosodic features for audio. We used decision level fusion to integrate the information from the two channels and experimented using the SUM rule and a neural network as the integration functions. The results indicate that even a simple linear function such as the SUM rule achieves very good performance in audiovisual fusion. We also experimented with different combinations of cues with the most informative being the facial expressions and the spectral features. The best combination of cues is the integration of facial expressions, spectral and prosodic features when a neural network is used as the fusion method. When tested on 96 audiovisual sequences, depicting spontaneously displayed (as opposed to posed) laughter and speech episodes, in a person independent way the proposed audiovisual approach achieves over 90% recall rate and over 80% precision.

REFERENCES

• 1: J. A. Bachorowski, M. J. Smoski, and M. J. Owren. The acoustic features of human laughter. Journal-Acoustical Society of America, 110(1):1581--1597, 2001.
• 2: P. Boersma and D. Weenink. Praat: doing phonetics by computer (version 4.3.01) (www.praat.org)). Technical report, 2005.
• 3: Rui Cai , Lie Lu , Hong-Jiang Zhang , Lian-Hong Cai, Highlight sound effects detection in audio stream, Proceedings of the 2003 International Conference on Multimedia and Expo - Volume 3 (ICME '03), p.37-40, July 06-09, 2003
• 4: N. Campbell, H. Kashioka, and R. Ohara. No laughing matter. In European conference on speech communciation and technology:; Interspeech, pages 465--468, 2005.
• 5: T. F. Cootes , C. J. Taylor , D. H. Cooper , J. Graham, Active shape models—their training and application, Computer Vision and Image Understanding, v.61 n.1, p.38-59, Jan. 1995 [doi>10.1006/cviu.1995.1004]
• 6: S. Dupont and J. Luettin. Audio-visual speech modeling for continuous speech recognition. Ieee Transactions on Multimedia, 2(3):141--151, 2000.
• 7: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 8: D. Gonzalez-Jimenez and J. L. Alba-Castro. Toward pose-invariant 2-d face recognition through point distribution models and facial symmetry. IEEE Transactions on Information Forensics and Security, 2(3):413--429, 2007.
• 9: H. Hermansky. Perceptual linear predictive (plp) analysis of speech. Journal of the Acoustical Society of America, 87(4):1738--1752, 1990.
• 10: Akinori Ito , Xinyue Wang , Motoyuki Suzuki , Shozo Makino, Smile and Laughter Recognition using Speech Processing and Face Recognition from Conversation Video, Proceedings of the 2005 International Conference on Cyberworlds, p.437-444, November 23-25, 2005 [doi>10.1109/CW.2005.82]
• 11: L. Kennedy and D. Ellis. Laughter detection in meetings. In NIST ICASSP 2004 Meeting Recognition Workshop, 2004.
• 12: Andrea Lockerd , Florian Mueller Mueller, LAFCam: Leveraging affective feedback camcorder, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506490]
• 13: I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, and V. Karaiskos. The ami meeting corpus. In International conference on methods and techniques in behavioral research; Proceedings of measuring behaviour 2005, pages 137--140, 2005.
• 14: Maja Pantic , Alex Pentland , Anton Nijholt , Thomas Huang, Human computing and machine understanding of human behavior: a survey, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181044]
• 15: I. Patras and M. Pantic. Particle filtering with factorized likelihoods for tracking facial features. In International conference on automatic face and gesture recognition, pages 97--104, 2004.
• 16: S. Petridis and M. Pantic. Audiovisual discrimination between laughter and speech. In International Conference on Acoustics Speech and Signal Processing, pages 5117--5120, 2008.
• 17: G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the Ieee, 91(9):1306--1326, 2003.
• 18: B. Reuderink. Fusion for audio-visual laughter detection. MS Thesis, University of Twente, the Netherlands, 2007.
• 19: J. A. Russell, J. A. Bachorowski, and J. M. Fernandez-Dols. Facial and vocal expressions of emotion. Annual Review of Psychology, 54:329--349, 2003.
• 20: K. Scherer. Affect bursts. In S. van Goozen, N. van de Poll, and J. Sergeant, editors, Emotions: Essays on emotion theory, pages 161--193. 1994.
• 21: M. Schroder, D. Heylen, and I. Poggi. Perception of non-verbal emotional listener feedback. In R. Hoffmann and H. Mixdorff, editors, Speech prosody, pages 1--4, 2006.
• 22: Khiet P. Truong , David A. van Leeuwen, Automatic discrimination between laughter and speech, Speech Communication, v.49 n.2, p.144-158, February, 2007 [doi>10.1016/j.specom.2007.01.001]
• 23: Michel F. Valstar , Hatice Gunes , Maja Pantic, How to distinguish posed from spontaneous smiles using geometric features, Proceedings of the 9th international conference on Multimodal interfaces, November 12-15, 2007, Nagoya, Aichi, Japan [doi>10.1145/1322192.1322202]
• 24: Z. Zeng, M. Pantic, G. Roisman, and T. Huang. A survey of affect recognition methods: Audio, visual and spontaneous expressions. IEEE Trans. Pattern Analysis and Machine Intelligence, accepted for publication, 30, 2008.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications

Additional Classification:
  J. Computer Applications
  J.m MISCELLANEOUS

General Terms:
Algorithms

Keywords:
audiovisual data processing, laughter detection, non-linguistic information processing