Multiple feature fusion by subspace learning

ABSTRACT

Since the emergence of extensive multimedia data, feature fusion has been more and more important for image and video retrieval, indexing and annotation. Existing feature fusion techniques simply concatenate a pair of different features or use canonical correlation analysis based methods for joint dimensionality reduction in the feature space. However, how to fuse multiple features in a generalized way is still an open problem. In this paper, we reformulate the multiple feature fusion as a general subspace learning problem. The objective of the framework is to find a general linear subspace in which the cumulative pairwise canonical correlation between every pair of feature sets is maximized after the dimension normalization and subspace projection. The learned subspace couples dimensionality reduction and feature fusion together, which can be applied to both unsupervised and supervised learning cases. In the supervised case, the pairwise canonical correlations of feature sets within the same classes are also counted in the objective function for maximization. To better model the high-order feature structure and overcome the computational difficulty, the features extracted from the same pattern source are represented by a single 2D tensor. The tensor-based dimensionality reduction methods are used to further extract low-dimensional discriminative features from the fused feature ensemble. Extensive experiments on visual data classification demonstrate the effectiveness and robustness of the proposed methods.

REFERENCES

• 1: T. Ahonen, A. Hadid, and M. Pietikainen. Face recognition with local binary patterns. In European Conference on Computer Vision, pages 469--481, 2004.
• 2: Timo Ahonen , Abdenour Hadid , Matti Pietikainen, Face Description with Local Binary Patterns: Application to Face Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.12, p.2037-2041, December 2006 [doi>10.1109/TPAMI.2006.244]
• 3: M. Barker and W. Rayens. Partial least squares for discrimination. Journal of Chemometrics, 17(3):166--173, 2003.
• 4: Peter N. Belhumeur , João P. Hespanha , David J. Kriegman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.711-720, July 1997 [doi>10.1109/34.598228]
• 5: Anna Bosch , Andrew Zisserman , Xavier Munoz, Representing shape with a spatial pyramid kernel, Proceedings of the 6th ACM international conference on Image and video retrieval, p.401-408, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282340]
• 6: Hwann-Tzong Chen , Huang-Wei Chang , Tyng-Luh Liu, Local Discriminant Embedding and Its Variants, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.846-853, June 20-26, 2005 [doi>10.1109/CVPR.2005.216]
• 7: Navneet Dalal , Bill Triggs, Histograms of Oriented Gradients for Human Detection, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.886-893, June 20-26, 2005 [doi>10.1109/CVPR.2005.177]
• 8: N. Dalal and B. Triggs. Object detection using histograms of oriented gradients. In European Conference on Computer Vision, Workshop on Pascal VOC'06, 2006.
• 9: Y. Fang, T. Tan, and Y. Wang. Fusion of global and local features for face verification. In IEEE Conf. on ICPR, pages 382--385, 2002.
• 10: Y. Fu and T. S. Huang. Image classification using correlation tensor analysis. IEEE Trans. on Image Processing, 17(2):226--234, 2008.
• 11: Y. Fu, M. Liu, and T. Huang. Conformal embedding analysis with local graph modeling on the unit hypersphere. In IEEE Conf. on CVPR, workshop on Component Analysis, 2007.
• 12: Athinodoros S. Georghiades , Peter N. Belhumeur , David J. Kriegman, From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.643-660, June 2001 [doi>10.1109/34.927464]
• 13: T.-K. Kim, O. Arandjelovic, and R. Cipolla. Learning over sets using boosted manifold principal angles (bompa). In British Machine Vision Conference, pages 779--788, 2005.
• 14: Tae-Kyun Kim , Josef Kittler , Roberto Cipolla, Discriminative Learning and Recognition of Image Set Classes Using Canonical Correlations, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.6, p.1005-1018, June 2007 [doi>10.1109/TPAMI.2007.1037]
• 15: T.-K. Kim, S.-F. Wong, and R. Cipolla. Tensor canonical correlation analysis for action classification. In IEEE Conf. on CVPR, 2007.
• 16: Kuang-Chih Lee , Jeffrey Ho , David J. Kriegman, Acquiring Linear Subspaces for Face Recognition under Variable Lighting, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.5, p.684-698, May 2005 [doi>10.1109/TPAMI.2005.92]
• 17: M. Liu, Y. Fu, and T. S. Huang. An audio-visual fusion framework with joint dimensionality reduction. In IEEE Conf. on ICASSP, 2008.
• 18: Timo Ojala , Matti Pietikäinen , Topi Mäenpää, Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.7, p.971-987, July 2002 [doi>10.1109/TPAMI.2002.1017623]
• 19: P. Jonathon Phillips , Patrick J. Flynn , Todd Scruggs , Kevin W. Bowyer , Jin Chang , Kevin Hoffman , Joe Marques , Jaesik Min , William Worek, Overview of the Face Recognition Grand Challenge, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.947-954, June 20-26, 2005 [doi>10.1109/CVPR.2005.268]
• 20: K. Srinivasa Rao , A. N. Rajagopalan, A probabilistic fusion methodology for face recognition, EURASIP Journal on Applied Signal Processing, v.2005 n.1, p.2772-2787, 1 January 2005 [doi>10.1155/ASP.2005.2772]
• 21: M. Sargin, Y. Yemez, E. Erzin, and A. Tekalp. Audiovisual synchronization and fusion using canonical correlation analysis. IEEE Trans. on Multimedia, 9(7):1396--1403, 2007.
• 22: Terence Sim , Simon Baker , Maan Bsat, The CMU Pose, Illumination, and Expression Database, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.12, p.1615-1618, December 2003 [doi>10.1109/TPAMI.2003.1251154]
• 23: Q.-S. Sun, Z. Jin, P.-A. Heng, and D.-S. Xia. A novel feature fusion method based on partial least squares regression. Lecture Notes in Computer Science 3686, 3686/2005:268--277, 2005.
• 24: Q.-S. Sun, S.-G. Zeng, Y. Liu, P.-A. Heng, and D.-S. Xia. A new method of feature fusion and its application in image recognition. Pattern Recognition, 38(12):2437--2448, 2005.
• 25: X. Tan and B. Triggs. Enhanced local texture feature sets for face recognition under difficult lighting conditions. In IEEE Conf. on AMFG, pages 168--182, 2007.
• 26: M. Turk and A. Pentland. Face recognition using eigenfaces. In IEEE Conf. on CVPR, pages 586--591, 1991.
• 27: Paul Viola , Michael J. Jones, Robust Real-Time Face Detection, International Journal of Computer Vision, v.57 n.2, p.137-154, May 2004 [doi>10.1023/B:VISI.0000013087.49260.fb]
• 28: X. Wang and X. Tang. Using random subspace to combine multiple features for face recognition. In IEEE Conf. on FGR, pages 284--289, 2004.
• 29: Lior Wolf , Amnon Shashua, Learning over sets using kernel principal angles, The Journal of Machine Learning Research, 4, p.913-931, 12/1/2003
• 30: Shuicheng Yan , Dong Xu , Qiang Yang , Lei Zhang , Xiaoou Tang , Hong-Jiang Zhang, Discriminant Analysis with Tensor Representation, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 1, p.526-532, June 20-26, 2005 [doi>10.1109/CVPR.2005.131]
• 31: Shuicheng Yan , Dong Xu , Benyu Zhang , Hong-Jiang Zhang , Qiang Yang , Stephen Lin, Graph Embedding and Extensions: A General Framework for Dimensionality Reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.1, p.40-51, January 2007 [doi>10.1109/TPAMI.2007.12]
• 32: J. Yang, J.-Y. Yang, D. Zhang, and J.-F. Lu. Feature fusion: parallel strategy vs. serial strategy. Pattern Recognition, 36(6):1369--1381, 2003.
• 33: Jiali Zhao , Haitao Wang , Haibing Ren , Seok-Cheol Kee, LBP Discriminant Analysis for Face Verification, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, p.167, June 20-26, 2005 [doi>10.1109/CVPR.2005.580]
• 34: Xiaoli Zhou , Bir Bhanu, Feature Fusion of Face and Gait for Human Recognition at a Distance in Video, Proceedings of the 18th International Conference on Pattern Recognition, p.529-532, August 20-24, 2006 [doi>10.1109/ICPR.2006.556]
• 35: Qiang Zhu , Mei-Chen Yeh , Kwang-Ting Cheng , Shai Avidan, Fast Human Detection Using a Cascade of Histograms of Oriented Gradients, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1491-1498, June 17-22, 2006 [doi>10.1109/CVPR.2006.119]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Feature representation

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Projections
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Algorithms

Keywords:
canonical correlation, face recognition, feature fusion, image set, subspace learning, tensor