Visual language modeling for image classification

ABSTRACT

Although it has been studied for many years, image classification is still a challenging problem. In this paper, we propose a visual language modeling method for content-based image classification. It transforms each image into a matrix of visual words, and assumes that each visual word is conditionally dependent on its neighbors. For each image category, a visual language model is constructed using a set of training images, which captures both the co-occurrence and proximity information of visual words. According to how many neighbors are taken in consideration, three kinds of language models can be trained, including unigram, bigram and trigram, each of which corresponds to a different level of model complexity. Given a test image, its category is determined by estimating how likely it is generated under a specific category. Compared with traditional methods that are based on bag-of-words models, the proposed method can utilize the spatial correlation of visual words effectively in image classification. In addition, we propose to use the absent words, which refer to those appearing frequently in a category but not in the target image, to help image classification. Experimental results show that our method can achieve comparable accuracy while performing classification much more quickly.

REFERENCES

• 1: Bahl, L. R., Jelinek, F., and Mercer, R. L. A maximum likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1983.
• 2: Peter F. Brown , John Cocke , Stephen A. Della Pietra , Vincent J. Della Pietra , Fredrick Jelinek , John D. Lafferty , Robert L. Mercer , Paul S. Roossin, A statistical approach to machine translation, Computational Linguistics, v.16 n.2, p.79-85, June 1990
• 3: Mays, E., Damerau, F. J. and Mercer, R. L. Context-based spelling correction. IBM Natural Language ITL, 1990.
• 4: Chatterjee, S., Hadi, A. and Price, B. Simple Linear Regression. Regression Analysis by Example, 3rd ed. New York: Wiley, 2000.
• 5: Sivic, J., Russell, B., Efros, A., Zisserman, A. and Freeman, W. Discovering object categories in image collections. Technical Report A. I. Memo 2005--005, MIT, 2005.
• 6: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 7: Fergus, R., Perona, P. and Zisserman, A. Object class recognition by unsupervised scale-invariant learning. In Proceedings of the Computer Vision and Pattern Recognition (CVPR'03), 2003.
• 8: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12, p.178, June 27-July 02, 2004
• 9: Csurka, G., Bray, C., Dance, C. and Fan, L. Visual categorization with bags of keypoints. In Workshop on Statistical Learning in Computer Vision, (ECCV'04), 2004, 1--22.
• 10: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]
• 11: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 12: R. Fergus , L. Fei-Fei , P. Perona , A. Zisserman, Learning Object Categories from Google"s Image Search, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1816-1823, October 17-20, 2005 [doi>10.1109/ICCV.2005.142]
• 13: Raphael Maree , Pierre Geurts , Justus Piater , Louis Wehenkel, Random Subwindows for Robust Image Classification, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.34-40, June 20-26, 2005 [doi>10.1109/CVPR.2005.287]
• 14: Wang, B., Li, Z. W., Li, M. J. and Ma, W. Y. Large-Scale Duplicate Detection for Web Image Search. In Proceedings of IEEE International Conference on Multimedia & Expo (ICME'06), 2006.
• 15: Katz, S. M. Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, 35(3):400--401, 1987.
• 16: Matas, J., Chum, O., Urban, M. and Pajdla, T. Robust wide baseline stereo from maximally stable extremal regions. In Proceedings of The British Machine Vision Conference (BMVC'02), 2002, 384--393.
• 17: K. Mikolajczyk , C. Schmid, An Affine Invariant Interest Point Detector, Proceedings of the 7th European Conference on Computer Vision-Part I, p.128-142, May 28-31, 2002
• 18: Qing-Fang Zheng , Wei-Qiang Wang , Wen Gao, Effective and efficient object-based image retrieval using visual phrases, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180664]
• 19: Otluman, H. and Aboulnasr, T. Low Complexity 2-d Hidden Markov Model for Face Recognition. In Proceedings of International Symposium on Computer Architecture. (ISCAS'00), 2000.
• 20: Vailaya, A., Jain, A. K. and Zhang, H. J. On image classification: City images vs. landscapes. Pattern Recognition, Vol. 31, pp. 1921--1936, 1998.
• 21: Oded Maron , Tomás Lozano-Pérez, A framework for multiple-instance learning, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.570-576, July 1998, Denver, Colorado, United States
• 22: Jinbo Bi , Yixin Chen , James Z. Wang, A Sparse Support Vector Machine Approach to Region-Based Image Categorization, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.1121-1128, June 20-26, 2005 [doi>10.1109/CVPR.2005.48]
• 23: P. Quelhas , F. Monay , J.-M. Odobez , D. Gatica-Perez , T. Tuytelaars , L. Van Gool, Modeling Scenes with Local Descriptors and Latent Aspects, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.883-890, October 17-20, 2005 [doi>10.1109/ICCV.2005.152]
• 24: Martin Szummer , Rosalind W. Picard, Indoor-Outdoor Image Classification, Proceedings of the 1998 International Workshop on Content-Based Access of Image and Video Databases (CAIVD '98), p.42, January 03-03, 1998
• 25: Gorkani, M. M. and Picard, R. W. Texture orientation for sorting photos .at a glance'. In Proc. 12th Int. Conf. on Pattern Recognition (ICPR'94), 1994, 459--464.
• 26: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]
• 27: Peng, F. and Schuurmans, D. Combining Naive Bayes and n-Gram Language Models for Text Classification. In Proc. of The 25th European Conference on Information Retrieval Research (ECIR'03), 2003.
• 28: Clarkson, P. R. and Rosenfeld, R. Statistical Language Modeling Using the CMU-Cambridge Toolkit. In Proceedings ESCA Eurospeech, 1997.
• 29: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms, Experimentation

Keywords:
absent word criterion, image classification, visual language model