Combination of audio and lyrics features for genre classification in digital audio collections

ABSTRACT

In many areas multimedia technology has made its way into mainstream. In the case of digital audio this is manifested in numerous online music stores having turned into profitable businesses. The widespread user adaption of digital audio both on home computers and mobile players show the size of this market. Thus, ways to automatically process and handle the growing size of private and commercial collections become increasingly important; along goes a need to make music interpretable by computers. The most obvious representation of audio files is their sound - there are, however, more ways of describing a song, for instance its lyrics, which describe songs in terms of content words. Lyrics of music may be orthogonal to its sound, and differ greatly from other texts regarding their (rhyme) structure. Consequently, the exploitation of these properties has potential for typical music information retrieval tasks such as musical genre classification; so far, there is a lack of means to efficiently combine these modalities. In this paper, we present findings from investigating advanced lyrics features such as the frequency of certain rhyme patterns, several parts-of-speech features, and statistic features such as words per minute (WPM). We further analyse in how far a combination of these features with existing acoustic feature sets can be exploited for genre classification and provide experiments on two test collections.

REFERENCES

• 1: S. Baumann, T. Pohle, and S. Vembu. Towards a socio-cultural compatibility of mir systems. In Proceedings of the 5th International Conference of Music Information Retrieval (ISMIR'04), pages 460--465, Barcelona, Spain, October 10-14 2004.
• 2: E. Brochu, N. de Freitas, and K. Bao. The sound of an album cover: Probabilistic multimedia and IR. In C. M. Bishop and B. J. Frey, editors, Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics, Key West, FL, USA, January 3-6 2003.
• 3: W. B. Cavnar and J. M. Trenkle. N-gram-based text categorization. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval (SDAIR'94), pages 161--175, Las Vegas, USA, 1994.
• 4: J. Downie. Annual Review of Information Science and Technology, volume 37, chapter Music Information Retrieval, pages 295--340. Information Today, Medford, NJ, 2003.
• 5: Jonathan Foote, An overview of audio information retrieval, Multimedia Systems, v.7 n.1, p.2-10, Jan. 1999 [doi>10.1007/s005300050106]
• 6: Denny Iskandar , Ye Wang , Min-Yen Kan , Haizhou Li, Syllabic level automatic synchronization of music signals and text lyrics, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180777]
• 7: Peter Knees , Markus Schedl , Tim Pohle , Gerhard Widmer, An innovative three-dimensional user interface for exploring music collections enriched, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180652]
• 8: P. Knees, M. Schedl, and G. Widmer. Multiple lyrics alignment: Automatic retrieval of song lyrics. In Proceedings of 6th International Conference on Music Information Retrieval (ISMIR'05), pages 564--569, London, UK, September 11-15 2005.
• 9: T. Lidy and A. Rauber. Evaluation of feature extractors and psycho-acoustic transformations for music genre classification. In Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR'05), pages 34--41, London, UK, September 11-15 2005.
• 10: B. Logan, A. Kositsky, and P. Moreno. Semantic analysis of song lyrics. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'04), pages 827--830, Taipei, Taiwan, June 27-30 2004.
• 11: Jose P. G. Mahedero , Álvaro MartÍnez , Pedro Cano , Markus Koppenberger , Fabien Gouyon, Natural language processing of lyrics, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101255]
• 12: R. Mayer, R. Neumayer, and A. Rauber. Rhyme and style features for musical genre classification by song lyrics. In Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR'08), Philadelphia, PA, USA, September 14-18 2008. Accepted for publication.
• 13: R. Neumayer and A. Rauber. Integration of text and audio features for genre classification in music information retrieval. In Proceedings of the 29th European Conference on Information Retrieval (ECIR'07), pages 724--727, Rome, Italy, April 2-5 2007.
• 14: R. Neumayer and A. Rauber. Multi-modal music information retrieval - visualisation and evaluation of clusterings by both audio and lyrics. In Proceedings of the 8th Conference Recherche d'Information Assistée par Ordinateur (RIAO'07), Pittsburgh, PA, USA, May 29th - June 1 2007.
• 15: Nicola Orio, Music retrieval: a tutorial and review, Foundations and Trends in Information Retrieval, v.1 n.1, p.1-96, January 2006 [doi>10.1561/1500000002]
• 16: E. Pampalk, A. Flexer, and G. Widmer. Hierarchical organization and description of music collections at the artist level. In Research and Advanced Technology for Digital Libraries ECDL'05, pages 37--48, 2005.
• 17: Elias Pampalk , Andreas Rauber , Dieter Merkl, Content-based organization and visualization of music archives, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641121]
• 18: A. Rauber, E. Pampalk, and D. Merkl. Using psycho-acoustic models and self-organizing maps to create a hierarchical structuring of music by musical styles. In Proceedings of the 3rd International Symposium on Music Information Retrieval (ISMIR'02), pages 71--80, Paris, France, October 13-17 2002.
• 19: Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 20: George Tzanetakis , Perry Cook, MARSYAS: a framework for audio analysis, Organised Sound, v.4 n.3, p.169-175, December 1999 [doi>10.1017/S1355771800003071]
• 21: G. Tzanetakis and P. Cook. Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5):293--302, July 2002.
• 22: Yongwei Zhu , Kai Chen , Qibin Sun, Multimodal content-based structure analysis of karaoke music, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101293]
• 23: Hugo Fastl , Eberhard Zwicker, Psychoacoustics: Facts and Models, Springer-Verlag New York, Inc., Secaucus, NJ, 2006

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process; Retrieval models

General Terms:
Experimentation, Measurement

Keywords:
audio features, feature fusion, feature selection, genre classification, lyrics processing, supervised learning