A computation method for video segmentation utilizing the pleasure-arousal-dominance emotional information

ABSTRACT

Extracting video structures is important for video indexing and navigation in large digital video archives. It is usually achieved by video segmentation algorithms. Little research efforts has been invested on segmentation solutions that utilize the video's emotional content. These solutions not only have the potential of providing better performances than existing segmentation methods, but are also able to provide a more natural video segmentation with which viewers can associate with. The development of an affect-based segmentation solution faces many challenges, such as the dynamic and time evolving nature of a video's emotional content. This paper introduces a novel computation method for affect-based video segmentation. It is designed based on the Pleasure-Arousal-Dominance (P-A-D) emotion model[18], which in principle can represent a large number of emotions. This method consists of a P-A-D estimation stage and a segmentation stage. A P-A-D estimator based on the Dynamic Bayesian Networks (DBNs) is proposed for the first stage. A clustering-based algorithm that utilizes the video's P-A-D information is proposed for the second stage. Experimental results demonstrate the feasibility of the method.

REFERENCES

• 1: M. Bradley. Emotional memory: a dimensional analysis. Hillsdale, NJ: Lawrence Erlbaum, 1994.
• 2: Celoxica. RC200/203 Manual, 2005.
• 3: Bee Yong Chua , Guojun Lu, Improved Perceptual Tempo Detection of Music, Proceedings of the 11th International Multimedia Modelling Conference, p.316-321, January 12-14, 2005 [doi>10.1109/MMMC.2005.49]
• 4: Alberto Del Bimbo, Visual information retrieval, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 5: P. Ekman. Facial expression and emotion. American Psycho., 48(4):384--392, Apr. 1993.
• 6: J. J. Gross and R. W. Levenson. Emotion elicitation using films. Cog. and Emot., 9(1):87--108, Jan. 1995.
• 7: A. Hanjalic and R. Lagendijk. Automated high level segmentation for advanced video retrieval systems. IEEE Trans. Circuits Syst. Video Technol., 9(4):580--588, June 1999.
• 8: A. Hanjalic and L. Q. Xu. Affective content representation and modeling. IEEE Trans. Multimedia, 7(1):143--154, Feb. 2005.
• 9: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 10: A. Jaimes, T. Nagamine, J. Liu, K. Omura, and N. Sebe. Affective meeting video analysis. In IEEE Intl. Conf. on Multimedia and Expo, pages 1412--1415, July 2005.
• 11: Hang-Bong Kang, Analysis of scene context related with emotional events, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641072]
• 12: Hang-Bong Kang, Affective content detection using HMMs, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957066]
• 13: J. R. Kender , B. -L. Yeo, Video Scene Segmentation via Continuous Video Coherence, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.367, June 23-25, 1998
• 14: P. Lang. Perspectives on Anger and Emotion, pages 109--134. Lawrence Erlbaum Associates, 1993.
• 15: J. Laroche. Estimating tempo, swing and beat locations in audio recordings. In IEEE App. of Signal Process. to Audio and Acoust., pages 135--138, Oct.2001.
• 16: Michael S. Lew, Principles of visual information retrieval, Springer-Verlag, London, 2000
• 17: R. Lienhart, S. Pfeiffer, and W. Effelsberg. Scene determination based on video and audio features. In Intl. Conf. of Multimedia Syst.
• 18: A. Mehrabian. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psycho., 14(4):261--292, Dec. 1996.
• 19: Simon Moncrieff , Chitra Dorai , Svetha Venkatesh, Affect computing in film through sound energy dynamics, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500231]
• 20: K. Murphy. Dynamic bayesian networks, Nov. 2002.
• 21: C. E. Osgood, G. J. Suci, and P. H. Tannenbaum. The Measurement of Meaning. University of Illinois Press, 1967.
• 22: M. Pantic and L. J. M. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE, 91(9):1370--1390, Sept. 2003.
• 23: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 24: T. Pohle. Extraction of audio descriptors and their evaluation in music clasification tasks. Diploma thesis, University of Kaiserslautern, Kaiserslautern, Germany, Jan. 2005.
• 25: Yong Rui , Thomas S. Huang , Sharad Mehrotra, Constructing table-of-content for videos, Multimedia Systems, v.7 n.5, p.359-368, September 1999 [doi>10.1007/s005300050138]
• 26: P. Shaver, J. Schwartz, D. Kirson, and G. O'Connor. Emotions in Social Psycology: Key Readings in Social Psycology, pages 26--56. Psychology Press, 2001.
• 27: P. Valdez and A. Mehrabian. Effects of color on emotions. J. of Exp. Psycho., 124(4):394--409, Dec.1994.
• 28: J. Vendrig and M. Worring. Systematic evaluation of logical story unit segmentation. IEEE Trans. Multimedia, 4(4):492--499, Dec. 2002.
• 29: M. Xu, L. T. Chia, and J. Jin. Affective content analysis in comedy and horror videos by audio emotional event detection. In IEEE Intl. Conf. on Multimedia and Expo, July 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Perceptual reasoning

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms, Experimentation, Performance

Keywords:
affective video content analysis, video segmentation