Heterogeneous multimedia data semantics mining using content and location context

ABSTRACT

Because it is very common that the heterogeneous multimedia data of the same semantics always exist jointly in many domain and application specific databases, it is very helpful to consider the location information when analyzing multimedia data. In this paper we propose a method of integrating the content and location context for multimedia data mining to enable the cross-media retrieval, by which the query examples and the returned results can be of different modalities, e.g. to query audios by an example of image. We construct a graph model by combing the multimedia content and location information. The graph model is then refined according to different strategies. The semantic correlations among multimedia data are calculated by learning the high-order neighborhood structure of the graph and the Multimedia Correlation Space is constructed in which the cross-media retrieval can be performed. We also propose different methods of Relevance Feedback to improve the search results. Experiments demonstrate the promise of the proposed method.

REFERENCES

• 1: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 2: Yueting Zhuang, Yi Yang, Fei Wu.: Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media Retrieval. IEEE Transactions on Multimedia 10(2): 221--229 2008.
• 3: Yi Yang, Yueting Zhuang, Fei Wu, Yunhe Pan.: Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media Retrieval. IEEE Transactions on Multimedia 10(3): 437--446 2008.
• 4: Hong Zhang , Yueting Zhuang , Fei Wu, Cross-modal correlation learning for clustering on image-audio dataset, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291290]
• 5: Cox, T. and Cox, M.: Multidimensional Scaling. Chapman & Hall, London, 1994.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation

Keywords:
cross media retrieval, multimedia content analysis, multimedia information retrieval, multimedia location context