Affective ranking of movie scenes using physiological signals and content analysis

ABSTRACT

In this paper, we propose an approach for affective ranking of movie scenes based on the emotions that are actually felt by spectators. Such a ranking can be used for characterizing the affective, or emotional, content of video clips. The ranking can for instance help determine which video clip from a database elicits, for a given user, the most joy. This in turn will permit video indexing and retrieval based on affective criteria corresponding to a personalized user affective profile.

A dataset of 64 different scenes from 8 movies was shown to eight participants. While watching, their physiological responses were recorded; namely, five peripheral physiological signals (GSR - galvanic skin resistance, EMG - electromyograms, blood pressure, respiration pattern, skin temperature) were acquired. After watching each scene, the participants were asked to self-assess their felt arousal and valence for that scene. In addition, movie scenes were analyzed in order to characterize each with various audio- and video-based features capturing the key elements of the events occurring within that scene.

Arousal and valence levels were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based audio and video features. We show that a correlation exists between arousal- and valence-based rankings provided by the spectator's self-assessments, and rankings obtained automatically from either physiological signals or audio-video features. This demonstrates the ability of using physiological responses of participants to characterize video scenes and to rank them according to their emotional content. This further shows that audio-visual features, either individually or combined, can fairly reliably be used to predict the spectator's felt emotion for a given scene. The results also confirm that participants exhibit different affective responses to movie scenes, which emphasizes the need for the emotional profiles to be user-dependant.

REFERENCES

• 1: K. Ansari-Asl, G. Chanel, and T. Pun, A channel selection method for EEG classification in emotion assessment based on synchronization likelihood, In Eusipco 2007, 15th Eur. Signal Proc. Conf., Poznan, Poland, September 2007.
• 2: A. Benoit, L. Bonnaud, A. Caplier, P. Ngo, L. Lawson, D. Trevisan, V. Levacik, C. Mancas, and G. Chanel, Multimodal focus attention and stress detection and feedback in an augmented driver simulator, In 3rd IFIP Conference on Artificial Intelligence Applications & Innovations (AIAI), Athens, Greece, June 2006.
• 3: Kirsten Boehner , Rogério DePaula , Paul Dourish , Phoebe Sengers, How emotion is made and measured, International Journal of Human-Computer Studies, v.65 n.4, p.275-291, April, 2007 [doi>10.1016/j.ijhcs.2006.11.016]
• 4: P. Boersma and D. Weenink, Praat: doing phonetics by computer (Version 5.0.05) {Computer program}. Retrieved January 19, 2008, from http://www.praat.org/.
• 5: Scott Brave , Clifford Nass , Kevin Hutchinson, Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent, International Journal of Human-Computer Studies, v.62 n.2, p.161-178, February 2005 [doi>10.1016/j.ijhcs.2004.11.002]
• 6: Ching Hau Chan , Gareth J. F. Jones, Affect-based indexing and retrieval of films, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101243]
• 7: G. Chanel G., K. Ansari-Asl, and T. Pun, Valence-arousal evaluation using physiological signals in an emotion recall paradigm, In 2007 IEEE SMC Int. Conf. on Systems, Man and Cybernetics, Smart cooperative systems and cybernetics: advancing knowledge and security for humanity, Montreal, Canada, October 2007.
• 8: G. Chanel, J. Kronegg, D. Grandjean, and T. Pun, Emotion assessment: Arousal evaluation using EEG's and peripheral physiological signals, Proc. Int. Workshop Multimedia Content Representation, Classification and Security (MRCS), B. Gunsel, A. K. Jain, A. M. Tekalp, B. Sankur, Eds., Lecture Notes in Computer Science, Vol. 4105, Springer, pages 530--537, Istanbul, Turkey, September 2006.
• 9: L. Chen, S. Gunduz, and M.T. Ozsu, Mixed type audio classification with support vector machine, In Int. Conf. on Multimedia and Expo, Toronto, Canada, July 2006.
• 10: R. R Cornelius, Theoretical approaches to emotion, Proc. Int. Speech Communication Association (ISCA) Workshop on Speech and Emotion, pages 3--10, Belfast, Northern Ireland, September 2000.
• 11: R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, Emotion recognition in human computer interaction, IEEE Signal Processing Magazine, 18(1):32--80, 2001.
• 12: G. B. Duchenne and R. A. Cuthbertson, The mechanism of human facial expression, Cambridge University Press, Cambridge, 1990.
• 13: P. Ekman, et al., Universals and cultural differences in the judgments of facial expressions of emotion, Journal of Personality and Social Psychology, 53(4):712--717, 1987.
• 14: A. Hanjalic and L. Q. Xu, Affective video content representation and modeling, IEEE Trans. Multimedia, 7(1):143--154, February 2005.
• 15: Jennifer A. Healey , Rosalind W. Picard, Wearable and automotive systems for affect recognition from physiology, Massachusetts Institute of Technology, 2000
• 16: Bruno Janvier , Eric Bruno , Thierry Pun , Stéphane Marchand-Maillet, Information-theoretic temporal segmentation of video and applications: multiscale keyframes selection and shot boundaries detection, Multimedia Tools and Applications, v.30 n.3, p.273-288, September 2006 [doi>10.1007/s11042-006-0026-2]
• 17: F. H. Kanfer, "Verbal Rate, Eyeblink, and Content in Structured Psychiatric Interviews," Journal of Abnormal and Social Psychology, 61(3): 341--347, 1960.
• 18: J. Kim, Emotion recognition from physiological measurement, In Humaine European Network of Excellence Workshop, Santorini, Greece, September 2004.
• 19: J. Kronegg, G. Chanel, S. Voloshynovskiy, and T. Pun, EEG--based synchronized brain--computer interfaces: A model for optimizing the number of mental tasks, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15(1):50--58, March 2007.
• 20: P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, Looking at pictures: affective, facial, visceral, and behavioral reactions, Psychophysiology, 30(3):261--273, 1993.
• 21: Dongge Li , Ishwar K. Sethi , Nevenka Dimitrova , Tom McGee, Classification of general audio data for content-based retrieval, Pattern Recognition Letters, v.22 n.5, p.533-544, April 2001 [doi>10.1016/S0167-8655(00)00119-7]
• 22: Lie Lu , Hao Jiang , HongJiang Zhang, A robust audio classification and segmentation method, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500173]
• 23: N. Moenne-Loccoz, OVAL: an object-based video access library to facilitate the development of content-based video retrieval systems. Technical report, Viper group - University of Geneva, 2004.
• 24: J. D. Morris, SAM: the self-assessment manikin, an efficient cross-cultural measurement of emotional response, Journal of Advertising Research,35(6):63--68 ,1995.
• 25: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 26: R. W. Picard and S. B. Daily, "Evaluating Affective Interactions: Alternatives to Asking What Users Feel", CHI Workshop on Evaluating Affective Interfaces: Innovative Approaches Portland: April 2005.
• 27: P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, International Journal of Psychophysiology, 61(1): 5--18, 2006.
• 28: Z. Rasheed, Y. Sheikh, and M. Shah, On the use of computable features for film classification, IEEE Transactions on Circuit and Systems for Video Technology, (11):52--64, 2005.
• 29: J. Rottenberg, R. D. Ray, and J. J Gross, Emotion elicitation using films. In: J. A. Coan & J. J. B. Allen (Eds.), The handbook of emotion elicitation and assessment. Oxford University Press, London, 2007.
• 30: J. Russell, A. Mehrabian, Evidence for a 3-factor theory of emotions, Journal of Research in Personality, 11(3):273--294, 1977.
• 31: K. Takahashi, Remarks on Emotion Recognition from Bio-Potential Signals, In 2nd International Conference on Autonomous Robots and Agents, Palmerston North, New Zealand, December 2004.
• 32: Michael E. Tipping, Sparse bayesian learning and the relevance vector machine, The Journal of Machine Learning Research, 1, p.211-244, 9/1/2001 [doi>10.1162/15324430152748236]
• 33: H. L. Wang, L. F. Cheong, Affective understanding in film, IEEE Transactions on Circuits and Systems for Video Technology, 16(6),:689--704, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process

General Terms:
Algorithms, Experimentation, Human Factors, Measurement

Keywords:
affective computing, affective personalization and ranking, emotion recognition and assessment, multimedia indexing and retrieval, physiological signals