Comprehensive query-dependent fusion using regression-on-folksonomies: a case study of multimodal music search

ABSTRACT

The combination of heterogeneous knowledge sources has been widely regarded as an effective approach to boost retrieval accuracy in many information retrieval domains. While various technologies have been recently developed for information retrieval, multimodal music search has not kept pace with the enormous growth of data on the Internet. In this paper, we study the problem of integrating multiple online information sources to conduct effective query dependent fusion (QDF) of multiple search experts for music retrieval. We have developed a novel framework to construct a knowledge space of users' information need from online folksonomy data. With this innovation, a large number of comprehensive queries can be automatically constructed to train a better generalized QDF system against unseen user queries. In addition, our framework models QDF problem by regression of the optimal combination strategy on a query. Distinguished from the previous approaches, the regression model of QDF (RQDF) offers superior modeling capability with less constraints and more efficient computation. To validate our approach, a large scale test collection has been collected from different online sources, such as Last.fm, Wikipedia, and YouTube. All test data will be released to the public for better research synergy in multimodal music search. Our performance study indicates that the accuracy, efficiency, and robustness of the multimodal music search can be improved significantly by the proposed folksonomy-RQDF approach. In addition, since no human involvement is required to collect training examples, our approach offers great feasibility and practicality in system development.

REFERENCES

• 1: A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In FOCS'06, 2006.
• 2: C. Chang and C. Lin. Libsvm: a library for support vector machines, 2001.
• 3: T.-S. Chua, S.-Y. Neo, K.-Y. Li, G. Wang, R. Shi, M. Zhao, and H. Xu. Trecvid 2004 search and feature extraction task by nus pris. In NIST TRECVID Workshop, 2004.
• 4: B. Cui, L. Liu, C. Pu, J. Shen, and K. L. Tan. Quest: querying music databases by acoustic and textual features. In ACM Multimedia'07, 2007.
• 5: C. V. Damme, M. Hepp, and K. Siorpaes. Folksontology: An integrated approach for turning folksonomies into ontologies. In the ESWC Workshop, 2007.
• 6: S. J. Downie. The music information retrieval evaluation exchange (mirex). In ISMIR'08, 2008.
• 7: N. R. Draper and H. Smith. Applied Regression Analysis. Wiley-Interscience, 1998.
• 8: I.-H. Kang and G. Kim. Query type classification for web document retrieval. In SIGIR '03, 2003.
• 9: L. Kennedy, S. F. Chang, and A. Natsev. Query-adaptive fusion for multimodal search. Proc. of the IEEE, 2008.
• 10: L. Kennedy, A. P. Natsev, and S. F. Chang. Automatic discovery of query-class-dependent models for multimodal search. In ACM Multimedia'05, 2005.
• 11: J. Kivinen, A. J. Smola, and R. C. Williamson. Online learning with kernels. IEEE Trans. on Signal Processing, 2004.
• 12: C. D. Manning, P. Raghavan, and H. SchÄutze. Introduction to Information Retrieval. Cambridge University Press, 2008.
• 13: Q. Mei, J. Jiang, H. Su, and C. Zhai. Search and tagging: Two sides of the same coin? Technical report, 2007.
• 14: P. Mika. Ontologies are us: A unified model of social networks and semantics. Web Semantics, 2007.
• 15: G. A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.
• 16: X. Olivares, M. Ciaramita, and R. van Zwol. Boosting image retrieval through aggregating search results based on visual annotations. In ACM Multimedia'08, 2008.
• 17: M. F. Porter. An algorithm for suffix stripping. Program, 1980.
• 18: S. E. Robertson, S. Walker, M. M. Beaulieu, and M. Gatford. Okapi at trec-4. In TREC-4, 1995.
• 19: S. S. Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-gradient solver for svm. In ICML'07, 2007.
• 20: B. SchÄolkopf and A. J. Smola. Learning with Kernels. Cambridge, MA: MIT Press, 2001.
• 21: J. A. Shaw and E. A. Fox. Combination of multiple searches. In TREC-2, 1994.
• 22: S. Shwartz and N. Srebro. SVM optimization: inverse dependence on training set size. In ICML'08, 2008.
• 23: A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR '06, 2006.
• 24: G. Tzanetakis and P. Cook. marsyas a framework for audio analysis. Organized Sound, 2000.
• 25: L. Xie, A. Natsev, and J. Tesic. Dynamic multimodal fusion in video search. In IEEE ICME'07, 2007.
• 26: R. Yan and A. G. Hauptmann. Probabilistic latent query analysis for combining multiple retrieval sources. In SIGIR'06, 2006.
• 27: R. Yan, J. Yang, and A. G. Hauptmann. Learning query-class dependent weights in automatic video retrieval. In ACM Multimedia'04, 2004.
• 28: B. Zhang, J. Shen, Q. Xiang, and Y. Wang. Compositemap: A novel framework for music similarity measure. In Proc. of ACM SIGIR, 2009.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.5 Sound and Music Computing
          Subjects: Systems

General Terms:
Algorithms, Design, Experimentation

Keywords:
folksonomy, multimodal search, music, query-dependent fusion