Towards optimal bag-of-features for object categorization and semantic video retrieval

ABSTRACT

Bag-of-features (BoF) deriving from local keypoints has recently appeared promising for object and scene classification. Whether BoF can naturally survive the challenges such as reliability and scalability of visual classification, nevertheless, remains uncertain due to various implementation choices. In this paper, we evaluate various factors which govern the performance of BoF. The factors include the choices of detector, kernel, vocabulary size and weighting scheme. We offer some practical insights in how to optimize the performance by choosing good keypoint detector and kernel. For the weighting scheme, we propose a novel soft-weighting method to assess the significance of a visual word to an image. We experimentally show that the proposed soft-weighting scheme can consistently offer better performance than other popular weighting methods. On both PASCAL-2005 and TRECVID-2006 datasets, our BoF setting generates competitive performance compared to the state-of-the-art techniques. We also show that the BoF is highly complementary to global features. By incorporating the BoF with color and texture features, an improvement of 50% is reported on TRECVID-2006 dataset.

REFERENCES

• 1: LSCOM lexicon definitions and annotations. In DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report #217-2006-3, 2006.
• 2: A. C. Berg and J. Malik. Geometric blur for template matching. In IEEE CVPR, 2001.
• 3: M. Campbell et al. IBM research trecvid-2006 video retrieval system. In TRECVID, 2006.
• 4: J. Cao et al. Intelligent multimedia group of Tsinghua university at trecvid 2006. In TRECVID, 2006.
• 5: C. C. Chang and C. J. Lin. LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm, 2001.
• 6: O. Chapelle, P. Haffner, and V. N. Vapnik. Support vector machines for histogram-based image classification. IEEE Trans. on NN, 10(5), 1999.
• 7: M. Everingham et al. The 2005 pascal visual object classes challenge. In LNAI, volume 3944, pages 117--176. Springer-Verlag, 2005.
• 8: Charless Fowlkes , Serge Belongie , Fan Chung , Jitendra Malik, Spectral Grouping Using the Nyström Method, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.2, p.214-225, January 2004 [doi>10.1109/TPAMI.2004.1262185]
• 9: A. G. Hauptmann et al. Multi-lingual broadcast news retrieval. In TRECVID, 2006.
• 10: D. Larlus, G. Dorko, and F. Jurie. Creation de vocabulaires visuels efficaces pour la categorisation d'images. In Reconnaissance des Formes et Intelligence Artificielle, 2006.
• 11: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 12: Tony Lindeberg, Feature Detection with Automatic Scale Selection, International Journal of Computer Vision, v.30 n.2, p.79-116, Nov. 1998 [doi>10.1023/A:1008045108935]
• 13: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 14: K. Mikolajczyk , T. Tuytelaars , C. Schmid , A. Zisserman , J. Matas , F. Schaffalitzky , T. Kadir , L. Van Gool, A Comparison of Affine Region Detectors, International Journal of Computer Vision, v.65 n.1-2, p.43-72, November 2005 [doi>10.1007/s11263-005-3848-x]
• 15: Krystian Mikolajczyk , Cordelia Schmid, Scale & Affine Invariant Interest Point Detectors, International Journal of Computer Vision, v.60 n.1, p.63-86, October 2004 [doi>10.1023/B:VISI.0000027790.02288.f2]
• 16: A. Agarwal, and B. Triggs. Hyperfeatures - multilevel local coding for visual recognition. In ECCV, 2006.
• 17: E. Nowak et al. Sampling strategies for bag-of-features image classification. In ECCV, 2006.
• 18: F. Odone et al. Building kernels from binary strings for image matching. IEEE Trans. on IP, 14(2), 2005.
• 19: S. Petrov et al. Detecting categories in news video using acoustic, speech, and image features. In TRECVID, 2006.
• 20: J. Platt. Probabilities for SV machines. In Advances in Large Margin Classifiers, pages 61--74, 2000.
• 21: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 22: C. G. M. Snoek et al. The mediamill trecvid 2006 semantic video search engine. In TRECVID, 2006.
• 23: TREC Video Retrieval Evaluation (TRECVID). http://www-nlpir.nist.gov/projects/trecvid/.
• 24: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 25: J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid. Local features and kernels for classification of texture and object categories: An in-depth study. In INRIA Technical Report RR-5737, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
bag-of-features, kernel, keypoint detector, object categorization, semantic video retrieval, soft-weighting