Hierarchical movie affective content analysis based on arousal and valence features

ABSTRACT

Emotional factors directly reflect audiences' attention, evaluation and memory. Affective contents analysis not only create an index for users to access their interested movie segments, but also provide feasible entry for video highlights. Most of the work focus on emotion type detection. Besides emotion type, emotion intensity is also a significant clue for users to find their interested content. For some film genres (Horror, Action, etc), the segments with high emotion intensity have the most possibilities to be video highlights. In this paper, we propose a hierarchical structure for emotion categories and analyze emotion intensity and emotion type by using arousal and valence related features hierarchically. Firstly, High, Medium and Low are detected as emotion intensity levels by using fuzzy c-mean clustering on arousal features. Fuzzy clustering provides a mathematical model to represent vagueness, which is close to human perception. After that, valence related features are used to detect emotion types (Anger, Sad, Fear, Happy and Neutral). Considering video is continuous time series data and the occurrence of a certain emotion is affected by recent emotional history, Hidden Markov Models (HMMs) are used to capture the context information. Experimental results shows the movie segments with high emotion intensity cover over 80% of the movie highlights in Horror and Action movies and the hierarchical method outperforms the one-step method on emotion type detection. Meanwhile, it is flexible for user to pick up their favorite affective content by choosing both emotion intensity levels and emotion types.

REFERENCES

• 1: Hang-Bong Kang, Affective content detection using HMMs, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957066]
• 3: A. Hanjalic and L. Q. Xu. A ffective video content representation and modeling. IEEE Transaction on multimedia, VOL 7, No. 1, Pages: 143--154, Feb., 2005.
• 4: H. L. Wang and L. F. Cheong Aff ective Understanding in Film. IEEE Transactions on Circuits and Systems for Video Technology, VOL. 16, No.6, Pages:689--704, June, 2006.
• 5: Z. Rasheed, Y. Sheikh and M. Shah. On the use of computable features for film classification. IEEE Transactions on Circuits and Systems for Video Technology, VOL. 15, Issue 1, Pages:52--64, Jan. 2005.
• 6: J. A. Russell and A. Mehrabian Evidence for a three-factor theory of emotions. Journal of Research in Personality, VOL. 11, Issue 3, Pages: 273--294, 1977.
• 7: P. Valdez and A. Mehrabian Eff ects of color on emotions. Journal of Experimental Psychology: General, VOL. 123, No. 4, Pages: 394--409, 1994.
• 8: C. Plantinga and G. M. Smith. Passionate Views: Film, Cognition, and Emotion. The Johns Hopkins University Press, 1999.
• 9: Z. Zeng, J. Tu, M. Liu, T. S. Huang, B. Pianfetti, D. Roth and S. Levinson. Audio-visual a ffective recognition. IEEE Transaction on Multimedia, VOL. 9, No. 2, Pages: 424--428, Feb., 2007.
• 10: Min Xu , N. C. Maddage , Changsheng Xu , M. Kankanhalli , Qi Tian, Creating audio keywords for event detection in soccer video, Proceedings of the 2003 International Conference on Multimedia and Expo, p.281-284, July 06-09, 2003
• 11: James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Kluwer Academic Publishers, Norwell, MA, 1981
• 12: Lawrence Rabiner , Biing-Hwang Juang, Fundamentals of speech recognition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993
• 23: Y. H. Chen, J. H. Kuo, W. T. Chu and J. L. Wu. Movie emotional event detection based on music mood and video tempo. 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods

General Terms:
Algorithms, Design, Experimentation

Keywords:
affective content, emotion intensity, emotion type, fuzzy clustering, hidden markov model, movie