Quantitative and visual analysis of the impact of music on perceived emotion of film

ABSTRACT

This article presents quantitative and visual methods for the analysis of the effect of music on emotion perceived in film. We discover strong, visible, and quantifiable trends in the effect of music on perceived emotion of film. We perform studies using both selected classical and composed music segments annotated with diverse emotions, paired with ambiguous film clips. We collect and analyze viewers' ratings of stress, activity, and dominance in the silent film, and film with various music soundtracks. The results are mapped onto a three-dimensional emotion space for visual and quantitative analysis. Aggregate scatter plots and center of mass results are presented using this emotion space. We find that the center of mass of the perceived emotion of film and music combined is consistently situated on a path between the center of mass of the musicalone response and that of the film-alone response. Regression analysis based on this trajectory observation results in coefficients with high R² values (R² = 0.675 for stress, R² = 0.817 for activity, and R² = 0.813 for dominance) for the first study with classical music selections, and lower R² values (R² = 0.199 for stress, R² = 0.405 for activity, and R² = 0.660 for dominance) for the second study with composed music. We conclude that continuous treatment of the two-dimensional emotion space provides a metric for comparing and assessing emotion ratings, and that spatial interpolation in this space provides a viable method for predicting the effect of music on perceived emotion of film.

REFERENCES

• 1: BOLIVAR, V. J., COHEN, A. J., AND FENTRESS, J. C. 1994. Semantic and formal congruency in music and motion pictures: Effect on theinterpretation of visual action. Psychomusicology 13, 28-59.
• 2: BULLERJAHN, C. AND GULDENRING, M. 1994. An empirical investigation of effects of film music using qualitative content analysis. Psychomusicology 13, 99-118.
• 3: COHEN, A. J. 2001. Music as a source of emotion in film. In Music and Emotion. P. N. Juslin and J. A. Sloboda (eds.), Oxford University Press, Oxford, 249-272.
• 4: GRIMM, M. 2005. Emotionen in der Sprache: Datenbankkonzepte und Intonationstranskription. Institut fur Nachrichtentechnik - Institut fur Automation und Robotik, Universitat Karlsruhe.
• 5: GRIMM, M. AND KROSCHEL, K. 2005. Evaluation of natural emotions using self assessment manikins. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop.
• 6: HURON, D. 2000. Perceptual and cognitive applications in music information retrieval. ISMIR.
• 7: JUSLIN, P. N. 2000. Cue utilization in communication of emotion in music performance: Relating performance to perception. J. Experimental Psychology 26, 1797-1813.
• 8: KEHREIN, R. 2002. The prosody of authentic emotions. In Proceedings of Speech Prosody Conference (Aix-en-Provence), 423--426.
• 9: LIPSCOMB, S. D. AND KENDALL, R. A. 1994. Perceptual judgment of the relationship between musical and visual components in film. Psychomusicology 13, 60-98.
• 10: LU, L., LIU, D., AND ZHANG, H. 2006. Automatic detection and tracking of music audio signals. IEEE Trans. Audio, Speech, and Lang. Proc. 14, 1, 5-18.
• 11: MARSHALL, S. K. AND COHEN, A. J. 1988. Effects of musical soundtrack on attitudes toward animated geometric fgures. Music Perception 6, 95-112.
• 12: OSGOOD, C. E., SUCI, G. J., AND TANNEBAUM, P.H. 1956. The Measurement of Meaning. University of Illinois Press, Urbana.
• 13: SIRIUS, G. AND CLARKE, E. F. 1994. The perception of audiovisual relationships: A preliminary study. Psychomusicology 13, 119-132.
• 14: SLOBODA, J. 1991. Music structure and emotional response: Some empirical findings. Psychology of Music 19, 110-120.
• 15: SMITH, J. 1999. Movie music as moving music: Emotion, cognition, and the film score. In Passionate Views: Film, Cognition, and Emotion, C. Pantiga and G. M. Smith (eds.), The Johns Hopkins University Press, Baltimore, MD,149-167.
• 16: THAYER, R. 1989. The Biopsychology of Mood and Arousal. Oxford University Press, New York.
• 17: YILDIRIM, S., BULUT M., LEE, C. M., KAZEMZADEH, A., BUSSO, C., DENG, Z., LEE, S., AND NARAYANAN, S. 2004. An acoustic study of emotions expressed in speech. In Proceedings of ICSLP.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.5 Sound and Music Computing

  J. Computer Applications
  J.5 ARTS AND HUMANITIES

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
audio, emotion rating, emotion space, film, music, video