More efficiency in multiple kernel learning

ABSTRACT

An efficient and general multiple kernel learning (MKL) algorithm has been recently proposed by Sonnenburg et al. (2006). This approach has opened new perspectives since it makes the MKL approach tractable for large-scale problems, by iteratively using existing support vector machine code. However, it turns out that this iterative algorithm needs several iterations before converging towards a reasonable solution. In this paper, we address the MKL problem through an adaptive 2-norm regularization formulation. Weights on each kernel matrix are included in the standard SVM empirical risk minimization problem with a l1 constraint to encourage sparsity. We propose an algorithm for solving this problem and provide an new insight on MKL algorithms based on block 1-norm regularization by showing that the two approaches are equivalent. Experimental results show that the resulting algorithm converges rapidly and its efficiency compares favorably to other MKL algorithms.

REFERENCES

• 1: Argyriou, A., Evgeniou, T., & Pontil, M. (2007). Convex multi-task feature learning (Technical Report).
• 2: Francis R. Bach , Gert R. G. Lanckriet , Michael I. Jordan, Multiple kernel learning, conic duality, and the SMO algorithm, Proceedings of the twenty-first international conference on Machine learning, p.6, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015424]
• 3: J. Frédéric Bonnans , Jean Charles Gilbert , Claude Lemaréchal , Claudia A. Sagastizábal, Numerical Optimization: Theoretical and Practical Aspects (Universitext), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 4: J. Frédéric Bonnans , Alexander Shapiro, Optimization Problems with Perturbations: A Guided Tour, SIAM Review, v.40 n.2, p.228-264, June 1998 [doi>10.1137/S0036144596302644]
• 5: Olivier Chapelle , Vladimir Vapnik , Olivier Bousquet , Sayan Mukherjee, Choosing Multiple Parameters for Support Vector Machines, Machine Learning, v.46 n.1-3, p.131-159, 2002 [doi>10.1023/A:1012450327387]
• 6: Dennis DeCoste , Kiri Wagstaff, Alpha seeding for support vector machines, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.345-349, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347165]
• 7: Grandvalet, Y. (1998). Least absolute shrinkage is equivalent to quadratic penalization. ICANN'98 (pp. 201--206). Springer.
• 8: Grandvalet, Y., & Canu, S. (2003). Adaptive scaling for feature selection in svms. Advances in Neural Information Processing Systems. MIT Press.
• 9: Gert R. G. Lanckriet , Tijl De Bie , Nello Cristianini , Michael I. Jordan , William Stafford Noble, A statistical framework for genomic data fusion, Bioinformatics, v.20 n.16, p.2626-2635, November 2004 [doi>10.1093/bioinformatics/bth294]
• 10: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 11: Claude Lemaréchal , Claudia Sagastizábal, Practical Aspects of the Moreau--Yosida Regularization: Theoretical Preliminaries, SIAM Journal on Optimization, v.7 n.2, p.367-385, 1997 [doi>10.1137/S1052623494267127]
• 12: Charles A. Micchelli , Massimiliano Pontil, Learning the Kernel Function via Regularization, The Journal of Machine Learning Research, 6, p.1099-1125, 12/1/2005
• 13: Scholkopf, B., & Smola, A. (2001). Learning with kernels. MIT Press.
• 14: Sören Sonnenburg , Gunnar Rätsch , Christin Schäfer , Bernhard Schölkopf, Large Scale Multiple Kernel Learning, The Journal of Machine Learning Research, 7, p.1531-1565, 12/1/2006
• 15: Wahba, G. (1990). Spline models for observational data. Series in Applied Mathematics, Vol. 59, SIAM.