Enhancing image annotation by integrating concept ontology and text-based bayesian learning model

ABSTRACT

Automatic image annotation (AIA) has been a hot research topic in recent years since it can be used to support concept-based image retrieval. However, most existing AIA models depend heavily on the availability of a large number of labeled training samples, which require significant human labeling efforts. In this paper, we propose a novel learning framework which integrates text-based Bayesian model (TBM) and concept ontology to effectively expand the training set of each concept class without the need of additional human labeling efforts or collecting additional training images from other data sources. The basic idea lies in exploiting the text information from training set to provide additional effective annotations for training images so that training data for each concept class can be augmented. In this study we employ Bayesian Hierarchical Multinomial Mixture Models (BHMMMs) as our baseline AIA model. By combining additional annotations obtained from TBM into each concept class in the training phase, the performance of BHMMMs can be significantly improved on Corel image dataset with 263 testing concepts as compared to the state-of-the-art AIA models under the same experimental configurations.

REFERENCES

• 1: K. Barnard, P. Duygulu and D. Forsyth, "Clustering Art", In Proc. Of IEEE Computer Vision and Pattern Recognition, 2001.
• 2: Gustavo Carneiro , Nuno Vasconcelos, Formulating Semantic Image Annotation as a Supervised Learning Problem, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.163-168, June 20-26, 2005 [doi>10.1109/CVPR.2005.164]
• 3: Jianping Fan , Hangzai Luo , Yuli Gao, Learning the Semantics of Images by Using Unlabeled Samples, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.704-710, June 20-26, 2005 [doi>10.1109/CVPR.2005.207]
• 4: Huamin Feng , Rui Shi , Tat-Seng Chua, A bootstrapping framework for annotating and retrieving WWW images, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027748]
• 5: S. L. Feng, R. Manmatha and V. Lavrenko, "Multiple Bernoulli Relevance Models for Image and Video Annotation", Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR'04.
• 6: S. Gao, D.-H. Wang and C.-H. Lee, "Automatic Image Annotation through Multi-Topic Text Categorization", Proc. ICASSP, Toulouse, France, May 2006.
• 7: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 8: V. Lavrenko, R. Manmatha and J. Jeon, "A Model for Learning the Semantics of Pictures", NIPS, 2003.
• 9: G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross and K. J. Miller, "Introduction to WordNet: an on-line lexical database", Intl. Jour. Of Lexicography, pp. 235--244, 1990.
• 10: J. Novovicova and A. Malik, "Application of Multinomial Mixture Model to Text Classification", Pattern Recognition and Image Analysis, pp. 646--653, 2003.
• 11: Munirathnam Srikanth , Joshua Varner , Mitchell Bowden , Dan Moldovan, Exploiting ontologies for automatic image annotation, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076128]
• 12: R. Shi, T. S. Chua, C. H. Lee and S. Gao, "Bayesian Learning of Hierarchical Multinomial Mixture Models of Concepts for Automatic Image Annotation", In Proc. of CIVR'06, pp. 102--112, Arizona, United States, 2006.
• 13: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 14: R. Yan, and A. G. Hauptmann, "Multi-class Active Learning for Video Semantic Feature Extraction", In Proc. of ICME'04, pp. 69--72, 2004.
• 15: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Computer vision

General Terms:
Algorithms, Human Factors, Performance

Keywords:
MAP, MLE, automatic image annotation, mixture model