Language modeling for bag-of-visual words image categorization

ABSTRACT

In this paper, we propose two ways of improving image classification based on bag-of-words representation [25]. Two shortcomings of this representation are the loss of the spatial information of visual words and the presence of noisy visual words due to the coarseness of the vocabulary building process. On the one hand, we propose a new representation of images that goes further in the analogy with textual data: visual sentences, that allows us to "read" visual words in a certain order, as in the case of text. We can therefore consider simple spatial relations between words. We also present a new image classification scheme that exploits these relations. It is based on the use of language models, a very popular tool from speech and text analysis communities. On the other hand, we propose new techniques to eliminate useless words, one based on geometric properties of the keypoints, the other on the use of probabilistic Latent Semantic Analysis (pLSA). Experiments show that our techniques can significantly improve image classification, compared to a classical Support Vector Machine-based classification.

REFERENCES

• 1: J. Bai and J.-Y. Nie. Using language models for text classification. In Proceedings of the Asia Information Retrieval Symposium, Beijing, China, Oct 2004.
• 2: A. Bosch, A. Zisserman, and X. Munoz. Scene classification via pLSA. In Proceedings of ECCV, 2006.
• 3: A. Bosch, A. Zisserman, and X. Munoz. Image classification using random forests and ferns. In Proceedings of ICCV, 2007.
• 4: G. Carneiro and A. Jepson. Flexible spatial models for grouping local image features. Proceedings of CVPR, 2:II--747--II--754 Vol.2, 27 Jun-2 Jul 2004.
• 5: W. B. Cavnar and J. M. Trenkle. N-gram-based text categorization. In Proceedings of the Symposium on Document Analysis and Information Retrieval, pages 161--175, Las Vegas, US, 1994.
• 6: S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. Technical report, Cambridge, MA, August 1998.
• 7: P. Clarkson and R. Rosenfeld. Statistical language modeling using the CMU-Cambridge toolkit. In Proceedings of the Eurospeech Conference, pages 2707--2710, Rhodes, Greece, 1997.
• 8: G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray. Visual categorization with bags of keypoints. In ECCV: Workshop on Statistical Learning in Computer Vision, Prague, Czech Republic, May 2004.
• 9: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12, p.178, June 27-July 02, 2004
• 10: S. Gao, D.-H. Wang, and C.-H. Lee. Automatic image annotation through multi-topic text categorization. volume 2, pages II--II, 2006.
• 11: D. Gokalp and S. Aksoy. Scene classification using bag-of-regions representations. In Proceedings of CVPR, pages 1--8, 2007.
• 12: Thomas Hofmann, Unsupervised learning by probabilistic latent semantic analysis, Machine Learning, v.42 n.1/2, p.177-196, Jan. 2001 [doi>10.1023/A:1007617005950]
• 13: M. Jamieson, A. Fazly, S. Dickinson, S. Stevenson, and S. Wachsmuth. Learning structured appearance models from captioned images of cluttered scenes. Oct 2007.
• 14: I. Jolliffe. Principal Component Analysis. Springer, 2002.
• 15: D. Larlus, G. Dorkó, and F. Jurie. Création de vocabulaires visuels efficaces pour la catégorisation d'images. In Reconnaissance des Formes et Intelligence Artificielle, 2006.
• 16: D. Larlus and F. Jurie. Latent mixture vocabularies for object categorization. In Proceedings of the British Machine Vision Conference, 2006.
• 17: K. Mc Donald. Discrete Language Models for Video Retrieval. PhD thesis, School of Computing, Dublin City University, September 2005.
• 18: Krystian Mikolajczyk , Cordelia Schmid, A Performance Evaluation of Local Descriptors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.10, p.1615-1630, October 2005 [doi>10.1109/TPAMI.2005.188]
• 19: K. Mikolajczyk , T. Tuytelaars , C. Schmid , A. Zisserman , J. Matas , F. Schaffalitzky , T. Kadir , L. Van Gool, A Comparison of Affine Region Detectors, International Journal of Computer Vision, v.65 n.1-2, p.43-72, November 2005 [doi>10.1007/s11263-005-3848-x]
• 20: F. Moosmann, B. Triggs, and F. Jurie. Randomized clustering forests for building fast and discriminative visual vocabularies. In Neural Information Processing Systems, Nov 2006.
• 21: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 22: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 23: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 24: Josef Sivic , Bryan C. Russell , Alexei A. Efros , Andrew Zisserman , William T. Freeman, Discovering Objects and their Localization in Images, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.370-377, October 17-20, 2005 [doi>10.1109/ICCV.2005.77]
• 25: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 26: Lei Wu , Mingjing Li , Zhiwei Li , Wei-Ying Ma , Nenghai Yu, Visual language modeling for image classification, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290101]
• 27: Jun Yang , Yu-Gang Jiang , Alexander G. Hauptmann , Chong-Wah Ngo, Evaluating bag-of-visual-words representations in scene classification, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290111]
• 28: J. Yuan, Y. Wu, and M. Yang. Discovery of collocation patterns: from visual words to visual phrases. In Proceedings of CVPR, pages 1--8, Jun 2007.
• 29: J. Zhang , M. Marszałek , S. Lazebnik , C. Schmid, Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, International Journal of Computer Vision, v.73 n.2, p.213-238, June 2007 [doi>10.1007/s11263-006-9794-4]
• 30: Qing-Fang Zheng , Wei-Qiang Wang , Wen Gao, Effective and efficient object-based image retrieval using visual phrases, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180664]
• 31: Lei Zhu , Al Bing Rao , Aldong Zhang, Theory of keyblock-based image retrieval, ACM Transactions on Information Systems (TOIS), v.20 n.2, p.224-257, April 2002 [doi>10.1145/506309.506313]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language models

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
        I.4.10 Image Representation

General Terms:
Design, Experimentation

Keywords:
bag-of-visual words, image categorization, language modeling, pLSA, support vector machines