Large-scale multimodal semantic concept detection for consumer video

ABSTRACT

In this paper we present a systematic study of automatic classification of consumer videos into a large set of diverse semantic concept classes, which have been carefully selected based on user studies and extensively annotated over 1300+ videos from real users. Our goals are to assess the state of the art of multimedia analytics (including both audio and visual analysis) in consumer video classification and to discover new research opportunities. We investigated several statistical approaches built upon global/local visual features, audio features, and audio-visual combinations. Three multi-modal fusion frameworks (ensemble, context fusion, and joint boosting) are also evaluated. Experiment results show that visual and audio models perform best for different sets of concepts. Both provide significant contributions to multimodal fusion, via expansion of the classifier pool for context fusion and the feature bases for feature sharing. The fused multimodal models are shown to significantly reduce the detection errors (compared to single modality models), resulting in a promising accuracy of 83% over diverse concepts. To the best of our knowledge, this is the first work on systematic investigation of multimodal classification using a large-scale ontology and realistic video corpus.

REFERENCES

• 1: C.C. Chang and C.J. Lin. LIBSVM: a Library for Support Vector Machines. 2001, http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 2: S.F. Chang, et al. Columbia University TRECVID-2005 Video Search and High-Level Feature Extraction. In NIST TRECVID workshop, Gaithersburg, MD, 2005.
• 3: A. Amir, et al. IBM Research TRECVID-2004 Video Retrieval System. In NIST TRECVID 2004 Workshop, Gaithersburg, MD, 2004.
• 4: R.Fergus, P. Perona, A. Zisserman. Object class recognition by unsupervised scale-invariant learning. IEEE Proc. CVPR, 2003, pp. 264--271.
• 5: J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: a statistical view of boosting. Dept. Statistics, Stanford University Technical Report, 1998.
• 6: K. Grauman and T. Darrel. Approximate correspondences in high dimensions. Advances in NIPS. 2006.
• 7: W. Jiang, S.F. Chang, and A.C. Loui. Kernel sharing with joint boosting for multi-class concept detection. In CVPR Workshop on Semantic Learning Applications in Multimedia, Minneapolis, MN, 2007.
• 8: W. Jiang, S.F. Chang, and A.C. Loui. Context-based concept fusion with boosted conditional random fields. In IEEE Proc. ICASSP. vol.1, 2007, pp. 949--952.
• 9: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 10: Alexander Loui , Jiebo Luo , Shih-Fu Chang , Dan Ellis , Wei Jiang , Lyndon Kennedy , Keansub Lee , Akira Yanagawa, Kodak's consumer video benchmark data set: concept definition and annotation, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290117]
• 11: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 12: NIST. TREC Video Retrieval Evaluation (TRECVID). 2001-2006, http://www-nlpir.nist.gov/projects/trecvid/
• 13: A. Torralba, K. Murphy, and W. Freeman. Sharing features: effective boosting procedure for multi-class object detection. In Proc. CVPR, vol. 2, 2004, pp. 762--769.
• 14: A. Torralba, K. Murphy, and W. Freeman. Contextual models for object detection using boosted random fields. Advances in NIPS, 2004.
• 15: A. Yanagawa, et al. Columbia University's Baseline Detectors for 374 LSCOM Semantic Visual Concepts. Columbia University ADVENT Tech. Report # 222-2006-8, March 2007, http://www.ee.columbia.edu/dvmm/columbia374 .
• 16: A. Yanagawa, W. Hsu, and S.-F. Chang. Brief Descriptions of Visual Features for Baseline TRECVID Concept Detectors. Columbia University ADVENT Tech. Report #219-2006-5, July 2006.
• 17: Caltech 101 data sets, http://www.vision.caltech.edu/Image_Datasets/Caltech101
• 18: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Multimedia databases

General Terms:
Algorithms, Management, Performance

Keywords:
consumer video indexing, multimedia ontology, semantic classification, video classification