More generality in efficient multiple kernel learning

ABSTRACT

Recent advances in Multiple Kernel Learning (MKL) have positioned it as an attractive tool for tackling many supervised learning tasks. The development of efficient gradient descent based optimization schemes has made it possible to tackle large scale problems. Simultaneously, MKL based algorithms have achieved very good results on challenging real world applications. Yet, despite their successes, MKL approaches are limited in that they focus on learning a linear combination of given base kernels.

In this paper, we observe that existing MKL formulations can be extended to learn general kernel combinations subject to general regularization. This can be achieved while retaining all the efficiency of existing large scale optimization algorithms. To highlight the advantages of generalized kernel learning, we tackle feature selection problems on benchmark vision and UCI databases. It is demonstrated that the proposed formulation can lead to better results not only as compared to traditional MKL but also as compared to state-of-the-art wrapper and filter methods for feature selection.

REFERENCES

• 1: Galen Andrew , Jianfeng Gao, Scalable training of L¹-regularized log-linear models, Proceedings of the 24th international conference on Machine learning, p.33-40, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273501]
• 2: Argyriou, A., Micchelli, C. A., & Pontil, M. (2005). Learning convex combinations of continuously parameterized basic kernels. Proceedings of the Workshop on Computational Learning Theory (pp. 338--352).
• 3: Bach, F. R. (2008). Exploring large feature spaces with hierarchical multiple kernel learning. Advances in Neural Information Processing Systems (pp. 105--112).
• 4: Francis R. Bach , Gert R. G. Lanckriet , Michael I. Jordan, Multiple kernel learning, conic duality, and the SMO algorithm, Proceedings of the twenty-first international conference on Machine learning, p.6, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015424]
• 5: Shumeet Baluja , Henry A. Rowley, Boosting Sex Identification Performance, International Journal of Computer Vision, v.71 n.1, p.111-119, January 2007 [doi>10.1007/s11263-006-8910-9]
• 6: Jinbo Bi , Tong Zhang , Kristin P. Bennett, Column-generation boosting methods for mixture of kernels, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014113]
• 7: Antoni B. Chan , Nuno Vasconcelos , Gert R. G. Lanckriet, Direct convex relaxations of sparse SVM, Proceedings of the 24th international conference on Machine learning, p.145-153, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273515]
• 8: Olivier Chapelle , Vladimir Vapnik , Olivier Bousquet , Sayan Mukherjee, Choosing Multiple Parameters for Support Vector Machines, Machine Learning, v.46 n.1-3, p.131-159, 2002 [doi>10.1023/A:1012450327387]
• 9: Crammer, K., Keshet, J., & Singer, Y. (2002). Kernel design using boosting. Advances in Neural Information Processing Systems (pp. 537--544).
• 10: Cristianini, N., Shawe-Taylor, J., Elisseeff, A., & Kandola, J. (2001). On kernel-target alignment. Advances in Neural Information Processing Systems (pp. 367--373).
• 11: Danskin, J. M. (1967). The theorey of max-min and its applications to weapons allocation problems.
• 12: Fung, G., & Mangasarian, O. L. (2002). A feature selection newton method for support vector machine classification (Technical Report 02-03). Univ. of Wisconsin.
• 13: Kloft, M., Brefeld, U., Laskov, P., & Sonnenburg, S. (2008). Non-sparse Multiple Kernel Learning. NIPS Workshop on Kernel Learning.
• 14: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 15: Baback Moghaddam , Ming-Hsuan Yang, Learning Gender with Support Faces, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.707-711, May 2002 [doi>10.1109/34.1000244]
• 16: Cheng Soon Ong , Alexander J. Smola , Robert C. Williamson, Learning the Kernel with Hyperkernels, The Journal of Machine Learning Research, 6, p.1043-1071, 12/1/2005
• 17: Rakotomamonjy, A., Bach, F., Grandvalet, Y., & Canu, S. (2008). Simplemkl. Journal of Machine Learning Research, 9, 2491--2521.
• 18: Le Song , Alex Smola , Arthur Gretton , Karsten M. Borgwardt , Justin Bedo, Supervised feature selection via dependence estimation, Proceedings of the 24th international conference on Machine learning, p.823-830, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273600]
• 19: Sören Sonnenburg , Gunnar Rätsch , Christin Schäfer , Bernhard Schölkopf, Large Scale Multiple Kernel Learning, The Journal of Machine Learning Research, 7, p.1531-1565, 12/1/2006
• 20: Varma, M., & Ray, D. (2007). Learning the discriminative power-invariance trade-off. Proceedings of the International Conference on Computer Vision.
• 21: Alexander Zien , Cheng Soon Ong, Multiclass multiple kernel learning, Proceedings of the 24th international conference on Machine learning, p.1191-1198, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273646]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.6 Optimization

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory