Active post-refined multimodality video semantic concept detection with tensor representation

ABSTRACT

In this paper, we resolve the problem of multi-modality video representation and semantic concept detection. Interaction and integration of multi-modality media types such as visual, audio and textual data in video are essential to video semantic analysis. Traditionally, videos are represented as vectors in the Euclidean space. Many learning algorithms are then taken to these vectors in a high dimensional space for dimension reduction, classification, clustering and so on. However, the multiple modalities in video not only have their own properties, but also have correlations among them; whereas the simple vector representation weakens the power of these relatively independent modalities and even ignores their relations to some extent. In this paper, we introduce a higher-order tensor framework for video analysis, in which we represent image, video and text three modalities in video shots as data points by the 3rd-order tensor called tensorshots. We propose a novel dimension reduction method that explicitly considers the manifold structure of the tensor space from multimodal media data which is temporal associated co-occurrence and then detect video semantic concepts through powerful classifiers which take tensor as input. Our algorithm preserves the intrinsic structure of the submanifold where tensorshots are sampled, and is also able to map out-of-sample data points directly. Moreover we apply an active learning based contextual and temporal post-refining strategy to enhance detection accuracy. Experiment results show that our method improves the performance of video semantic concept detection.

REFERENCES

• 1: N. Babaguchi, Y. Kawai, T. Kitahashi. Event based indexing of broadcast sports video by intermodal collaboration. In IEEE Transactions on Multimedia, 2002
• 2: Cees G. M. Snoek, Marcel Worring. Multimedia event-based video indexing using time intervals. In IEEE Transactions on Multimedia, 2005
• 3: Yanan Liu, Fei Wu. Video semantic concept detection using multi-modality subspace correlation propagation. In 13th Int. Multimedia Modeling Conf. (mmm2007), 2006.
• 4: M. A. O. Vasilescu , Demetri Terzopoulos, Multilinear Analysis of Image Ensembles: TensorFaces, Proceedings of the 7th European Conference on Computer Vision-Part I, p.447-460, May 28-31, 2002
• 5: Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in Neural Information Processing Systems (NIPS2002), MIT Press, Cambridge 585--591.
• 6: Xiaofei He, and Partha Niyogi. Locality preserving projections. Advances in Neural Information Processing Systems (NIPS2003).
• 7: Xiaofei He , Deng Cai , Haifeng Liu , Jiawei Han, Image clustering with tensor representation, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101169]
• 8: Lieven De Lathauwer , Bart De Moor , Joos Vandewalle, A Multilinear Singular Value Decomposition, SIAM Journal on Matrix Analysis and Applications, v.21 n.4, p.1253-1278, March – May 2000 [doi>10.1137/S0895479896305696]
• 9: Dacheng Tao , Xuelong Li , Xindong Wu , Weiming Hu , Stephen J. Maybank, Supervised tensor learning, Knowledge and Information Systems, v.13 n.1, p.1-42, September 2007 [doi>10.1007/s10115-006-0050-6]
• 10: Christopher J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, v.2 n.2, p.121-167, June 1998 [doi>10.1023/A:1009715923555]
• 11: Guo-Jun Qi , Xian-Sheng Hua , Yong Rui , Jinhui Tang , Tao Mei , Hong-Jiang Zhang, Correlative multi-label video annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291245]
• 12: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 13: Jun Yang , Alexander G. Hauptmann, Exploring temporal consistency for video analysis and retrieval, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178685]
• 14: I. T. Jolliffe. Principal Component Analysis. Springer, New York, 2nd edition, 2002.
• 15: T. Cox and M. Cox. Multidimensional Scaling. Chapman & Hall, London, 1994.
• 16: Sam T. Roweis, Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, Vol.290, 2323--2326, 2000.
• 17: Joshua B. Tenenbaum, Vin de Silva, John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, Vol.290, 2319--2323, 2000.
• 18: K. Q. Weinberger, B. D. Packer, and L. K. Saul. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In Proceedings of the Tenth International Workshop on AI and Statistics (AISTATS-05), Barbados, WI, 2005.
• 19: L. K. Saul, K. Q. Weinberger, Fei Sha, Jihun Ham, and Daniel D. Lee. Spectral Methods for Dimensionality Reduction - Semisupervised Learning. MIT Press, Cambridge, MA, 2006.
• 20: M. A. Turk and A. P. Pentland. Face recognition using eigenfaces. Computer Vision and Pattern Recognition, 586--591, 1991.
• 21: L. Itti, C. Koch and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 25, no 9, 1075--1088, 2003.
• 22: Dacheng Tao , Xuelong Li , Stephen J. Maybank , Xindong Wu, Human Carrying Status in Visual Surveillance, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1670-1677, June 17-22, 2006 [doi>10.1109/CVPR.2006.138]
• 23: Lieven De Lathauwer , Joséphine Castaing, Tensor-based techniques for the blind separation of DS-CDMA signals, Signal Processing, v.87 n.2, p.322-336, February, 2007 [doi>10.1016/j.sigpro.2005.12.015]
• 24: Ning Liu , Benyu Zhang , Jun Yan , Zheng Chen , Wenyin Liu , Fengshan Bai , Leefeng Chien, Text Representation: From Vector to Tensor, Proceedings of the Fifth IEEE International Conference on Data Mining, p.725-728, November 27-30, 2005 [doi>10.1109/ICDM.2005.144]
• 25: Rong Yan , Jun Yang , Alexander G. Hauptmann, Learning query-class dependent weights in automatic video retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027661]
• 26: Apostol (Paul) Natsev , Milind R. Naphade , Jelena TešiĆ, Learning the semantics of multimedia queries and concepts from a small number of examples, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101288]
• 27: Simon Tong , Daphne Koller, Support vector machine active learning with applications to text classification, The Journal of Machine Learning Research, 2, p.45-66, 3/1/2002 [doi>10.1162/153244302760185243]
• 28: Ming-yu Chen , Michael Christel , Alexander Hauptmann , Howard Wactlar, Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101342]
• 29: Lieven De Lathauwer. Signal Processing based on Multilinear Algebra. Ph.D. Thesis, September 1997.
• 30: S. T. Dumais , G. W. Furnas , T. K. Landauer , S. Deerwester , R. Harshman, Using latent semantic analysis to improve access to textual information, Proceedings of the SIGCHI conference on Human factors in computing systems, p.281-285, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57214]
• 31: Xiaofei He, Deng Cai, and Partha Niyogi. Tensor subspace analysis. Advances in Neural Information Processing Systems (NIPS2005).
• 32: Fan Rong K. Chung. Spectral Graph Theory, volume 92 of Regional Conference Series in Mathematics. 1997.
• 33: Brett W. Bader and Tamara G. Kolda. MATLAB Tensor Classes for Fast Algorithm Prototyping. Technical Report SAND2004-5187, Sandia National Laboratories, October 2004.
• 34: Brett. W. Bader and Tamara G. Kolda. Efficient MATLAB Computations with Sparse and Factored Tensors. Technical Report SAND02006-7592, Sandia National Laboratories, December 2006.
• 35: Zheng-Jun Zha , Tao Mei , Zengfu Wang , Xian-Sheng Hua, Building a comprehensive ontology to refine video concept detection, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290114]
• 36: Y. Y. Yao. Information-theoretic measures for knowledge discovery and data mining. In Entropy Measure, Maximum Entropy Principle and Emerging Applications, pages 115--136. Springer, 2003.
• 37: TREVID. http://www-nlpir.nist.gov/projects/trevid/.
• 38: LSCOM lexicon definitions and annotations version 1.0. In DTO Challenge Workshop on Large Scale Concept Ontology for Multimedia, Columbia University ADVENT Technical Report 117-2006-3, 2006.
• 39: Hong Zhang , Yueting Zhuang , Fei Wu, Cross-modal correlation learning for clustering on image-audio dataset, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291290]
• 40: Yi Yang, Yueting Zhuang, Fei Wu, Yunhe Pan. Harmonizing hierarchical manifolds for multimedia document semantics understanding and cross-media retrieval. In IEEE Transactions on Multimedia, 10(3): 437--446, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Design, Measurement

Keywords:
active learning, contextual correlation, dimension reduction, hosvd, multi-modality video semantic concept detection, support tensor machines (stm), temporal associated cooccurrence (tac), temporal dependency, tensorshot