Automatic image annotation via local multi-label classification

ABSTRACT

As the consequence of semantic gap, visual similarity does not guarantee semantic similarity, which in general is conflicting with the inherent assumption of many generative-based image annotation methods. While discriminative learning approach had often been used to classify images into different semantic classes, its efficiency is often impaired by the problems of multi-labeling and large scale concept space typically encountered in practical image annotation tasks. In this paper, we explore solutions to the problems of large scale concept space learning and mismatch between semantic and visual space. To tackle the first problem, we explore the use of higher level semantic space with lower dimension by clustering correlated keywords into topics in the local neighborhood. The topics are used as lexis for assigning multiple labels for unlabeled images. To tackle the problem of semantic gap, we aim to reduce the bias between visual and semantic spaces by finding optimal margins in both spaces. In particular, we propose an iterative solution by alternately maximizing the sum of the margins to reduce the gap between visual similarity and semantic similarity. The experimental results on the ECCV2002 benchmark show that our method outperforms the state-of-the-art generative-based annotation method MBRM and discriminative-based ASVM-MIL by 9% and 11% in terms of F1 measure respectively.

REFERENCES

• 1: A.Yavlinsky, E.Schofield, and S.Ruger. Annotation using global features and robust nonparametric density estimation. CIVR, 2005.
• 2: K. Barnard and D. Forsyth. Learning the semantics of words and pictures. ICCV, pages 408--415, 2001.
• 3: James C. Bezdek , Richard J. Hathaway, Convergence of alternating optimization, Neural, Parallel & Scientific Computations, v.11 n.4, p.351-368, December 2003
• 4: Gustavo Carneiro , Antoni B. Chan , Pedro J. Moreno , Nuno Vasconcelos, Supervised Learning of Semantic Classes for Image Annotation and Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.394-410, March 2007 [doi>10.1109/TPAMI.2007.61]
• 5: S. Deerwester, S. Dumais, G. Furnas, L. T. K., and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, pages 391--407, 1990.
• 6: P. Duygulu , Kobus Barnard , J. F. G. de Freitas , David A. Forsyth, Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.97-112, May 28-31, 2002
• 7: Jianping Fan , Yuli Gao , Hangzai Luo, Hierarchical classification for automatic image annotation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277763]
• 8: S. Feng, R. Manmatha, and V. Lavrenko. Multiple bernoulli relevance models for image and video annotation. CVPR, pages 1002--1009, 2004.
• 9: D. A. Forsyth , M. M. Fleck, Body plans, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.678, June 17-19, 1997
• 10: Yuli Gao , Jianping Fan , Xiangyang Xue , Ramesh Jain, Automatic image annotation by incorporating feature hierarchy and boosting to scale up SVM classifiers, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180840]
• 11: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 12: Niels Haering , Zarina Myles , Niels da Vitoria Lobo, Locating Deciduous Trees, Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '97), p.18, June 20-20, 1997
• 13: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 14: Rong Jin , Joyce Y. Chai , Luo Si, Effective automatic image annotation via a coherent language model and active learning, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027732]
• 15: Yohan Jin , Latifur Khan , Lei Wang , Mamoun Awad, Image annotations by combining multiple evidence & wordNet, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101305]
• 16: Feng Kang , Rong Jin , Rahul Sukthankar, Correlated Label Propagation with Application to Multi-label Learning, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1719-1726, June 17-22, 2006 [doi>10.1109/CVPR.2006.90]
• 17: V. Lavrenko, R. Manmatha, and J. Jeon. A model for learning the semantics of pictures. NIPS, pages 553--560, 2004.
• 18: Jing Liu , Bin Wang , Mingjing Li , Zhiwei Li , Weiying Ma , Hanqing Lu , Songde Ma, Dual cross-media relevance model for image annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291380]
• 19: Yi Li , Linda G. Shapiro, Consistent Line Clusters for Building Recognition in CBIR, Proceedings of the 16 th International Conference on Pattern Recognition (ICPR'02) Volume 3, p.30952, August 11-15, 2002
• 20: Florent Monay , Daniel Gatica-Perez, PLSA-based image auto-annotation: constraining the latent space, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027608]
• 21: Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector quantizing images with words. In First International Workshop on Multimedia Intellegent Storage and Retrieval Management, 1999.
• 22: Milind Naphade , John R. Smith , Jelena Tesic , Shih-Fu Chang , Winston Hsu , Lyndon Kennedy , Alexander Hauptmann , Jon Curtis, Large-Scale Concept Ontology for Multimedia, IEEE MultiMedia, v.13 n.3, p.86-91, July 2006 [doi>10.1109/MMUL.2006.63]
• 23: Guo-Jun Qi , Xian-Sheng Hua , Yong Rui , Jinhui Tang , Tao Mei , Hong-Jiang Zhang, Correlative multi-label video annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291245]
• 24: R.Shi, T. Chua, C. lee, and S. Gao. Bayesian learning of hierarchical multinomial mixture models of concepts for automatic image annotation. CIVR, pages 102--112, 2006.
• 25: Steven C. H. Hoi , Michael R. Lyu , Rong Jin, Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval, Proceedings of the 21st International Conference on Data Engineering Workshops, p.1177, April 05-08, 2005 [doi>10.1109/ICDE.2005.233]
• 26: Bingjun Sun , Prasenjit Mitra , C. Lee Giles , John Yen , Hongyuan Zha, Topic segmentation with shared topic detection and alignment of multiple documents, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277778]
• 27: Martin Szummer , Rosalind W. Picard, Indoor-Outdoor Image Classification, Proceedings of the 1998 International Workshop on Content-Based Access of Image and Video Databases (CAIVD '98), p.42, January 03-03, 1998
• 28: Jiayu Tang , Paul H. Lewis, Using multiple segmentations for image auto-annotation, Proceedings of the 6th ACM international conference on Image and video retrieval, p.581-586, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282363]
• 29: A. Vailaya , A. Jain , H. J. Zhang, On Image Classification: City vs. Landscape, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.3, June 21-21, 1998
• 30: L. Xu, J. Neufeld, B. Larson, and D. Schuurmans. Maximum margin clustering. NIPS, 2004.
• 31: L. Xu and D. Schuurmans. Unsupervised and semi-supervised multi-class support vector machines. NIPS, 2006.
• 32: Changbo Yang , Ming Dong , Jing Hua, Region-based Image Annotation using Asymmetrical Support Vector Machine-based Multiple-Instance Learning, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2057-2063, June 17-22, 2006 [doi>10.1109/CVPR.2006.250]
• 33: Kai Zhang , Ivor W. Tsang , James T. Kwok, Maximum margin clustering made practical, Proceedings of the 24th international conference on Machine learning, p.1119-1126, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273637]
• 34: Min-Ling Zhang , Zhi-Hua Zhou, ML-KNN: A lazy learning approach to multi-label learning, Pattern Recognition, v.40 n.7, p.2038-2048, July, 2007 [doi>10.1016/j.patcog.2006.12.019]
• 35: Xiangdong Zhou , Mei Wang , Qi Zhang , Junqi Zhang , Baile Shi, Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching, Proceedings of the 6th ACM international conference on Image and video retrieval, p.25-32, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282284]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Performance

Keywords:
automatic image annotation, maximum margin clustering, multi-label classification