Dual cross-media relevance model for image annotation

ABSTRACT

Image annotation has been an active research topic in recent years due to its potential impact on both image understanding and web image retrieval. Existing relevance-model-based methods perform image annotation by maximizing the joint probability of images and words, which is calculated by the expectation over training images. However, the semantic gap and the dependence on training data restrict their performance and scalability. In this paper, a dual cross-media relevance model (DCMRM) is proposed for automatic image annotation, which estimates the joint probability by the expectation over words in a pre-defined lexicon. DCMRM involves two kinds of critical relations in image annotation. One is the word-to-image relation and the other is the word-to-word relation. Both relations can be estimated by using search techniques on the web data as well as available training data. Experiments conducted on the Corel dataset and a web image dataset demonstrate the effectiveness of the proposed model.

REFERENCES

• 1: Cilibrasi, R., Vitanyi, P. M. B. Automatic Extraction of Meaning from the Web, Proc. IEEE International Symp. Information Theory, USA, 2006.
• 2: Chang, E., et al. CBSA: Content-based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machines. CirSysVideo, 2003. 13(1): p. 26--38.
• 3: Cusano, C., G. Ciocca, and R. Schettini. Image Annotation Using SVM. Proceedings of Internet Imaging, Vol. SPIE 5304. 2004.
• 4: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 5: Feng S., Manmatha, R., and Laverenko V. Multiple Bernoulli Relevance Models for Image and Video Annotation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1002--1009, 2004.
• 6: Xin Fan , Xing Xie , Zhiwei Li , Mingjing Li , Wei-Ying Ma, Photo-to-search: using multimodal queries to search the web from mobile devices, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101851]
• 7: Goldstone R. Similarity, Interactive Activation and Mapping. Journal Experimental Psychology: Learning, Memory, and Cognition, 20, 1994, 3--28
• 8: http://blog.searchenginewatch.com/blog/050809-200323
• 9: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 10: Jiang, J., Conrath, D. Semantic Similarity based on Corpus Statistics and Lexical Taxonomy. Proceedings on International Conference on Research in Computational Linguistics, 1997.
• 11: Rong Jin , Joyce Y. Chai , Luo Si, Effective automatic image annotation via a coherent language model and active learning, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027732]
• 12: Yohan Jin , Latifur Khan , Lei Wang , Mamoun Awad, Image annotations by combining multiple evidence & wordNet, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101305]
• 13: Feng Kang , Rong Jin , Rahul Sukthankar, Correlated Label Propagation with Application to Multi-label Learning, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1719-1726, June 17-22, 2006 [doi>10.1109/CVPR.2006.90]
• 14: Lavrenko, V., Manmatha, R., and Jeon, J. A Model for Learning the Semantics of Pictures. Proceedings of Advance in Neutral Information Processing, 2003.
• 15: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 16: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 17: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 18: Changhu Wang , Feng Jing , Lei Zhang , Hong-Jiang Zhang, Image annotation refinement using random walk with restarts, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180774]
• 19: Xin-Jing Wang , Lei Zhang , Feng Jing , Wei-Ying Ma, AnnoSearch: Image Auto-Annotation by Search, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1483-1490, June 17-22, 2006 [doi>10.1109/CVPR.2006.58]
• 20: Yeh, T., Tollmar, K., Darrell, T. Searching the Web with Mobile Images for Location Recognition. International Conference on Computer Vision and Pattern Recognition, 2004, pp. 76--81.
• 21: Keiji Yanai , Kobus Barnard, Image region entropy: a measure of "visualness" of web images associated with one concept, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101241]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
image annotation, image retrieval, relevance model, word correlation