Automatic music video generation based on temporal pattern analysis

ABSTRACT

Music video (MV) is a short film meant to present a visual representation of a popular music song. In this paper, we present a system that automatically generates MV-like videos from personal home videos based on observations that generally there are obvious repetitive visual and aural patterns in MVs. Based on a set of video and music analysis algorithms, the automatic music video (AMV) generation system automatically extracts temporal structures of the video and music, as well as repetitive patterns in the music. And then, according to the structure and patterns, a set of highlight segments from the raw home video footage are selected, aiming at matching the visual content with the aural structure and pattern. And last, the output music video is rendered by connecting the selected highlight video segments with appropriate transition effects, accompanied with the music. Experiments show that the results are compelling and promising.

REFERENCES

• 1: Dwelle, T. Music Video 101. {eBook} http://www.timtv.com.
• 2: Jonathan Foote , Matthew Cooper , Andreas Girgensohn, Creating music videos using automatic media analysis, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641119]
• 3: Hua, X. S., et al. Optimization-Based Automated Home Video Editing System. IEEE Trans. on Circuits and Systems for Video Technology. Vol 14, No. 5, May 2004, 572--583.
• 4: Wang, M., et al. Repeating Pattern Discovery from Acoustic Musical Signals. Intl Conf. on Multimedia and Expo, 2004.
• 5: Yu-Fei Ma , Lie Lu , Hong-Jiang Zhang , Mingjing Li, A user attention model for video summarization, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641116]
• 6: John C. Platt, AutoAlbum: Clustering Digital Photographs using Probabilistic Model Merging, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'00), p.96, June 16-16, 2000
• 7: Whitley, D. A Genetic Algorithm Tutorial. Statistics and Computing, Vol. 4, 64--85, 1994.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Experimentation

Keywords:
music analysis, music video, optimization, video content analysis, video editing, video segmentation