Localized multiple kernel learning

ABSTRACT

Recently, instead of selecting a single kernel, multiple kernel learning (MKL) has been proposed which uses a convex combination of kernels, where the weight of each kernel is optimized during training. However, MKL assigns the same weight to a kernel over the whole input space. In this paper, we develop a localized multiple kernel learning (LMKL) algorithm using a gating model for selecting the appropriate kernel function locally. The localizing gating model and the kernel-based classifier are coupled and their optimization is done in a joint manner. Empirical results on ten benchmark and two bioinformatics data sets validate the applicability of our approach. LMKL achieves statistically similar accuracy results compared with MKL by storing fewer support vectors. LMKL can also combine multiple copies of the same kernel function localized in different parts. For example, LMKL with multiple linear kernels gives better accuracy results than using a single linear kernel on bioinformatics data sets.

REFERENCES

• 1: Ethem Alpaydin, Combined 5 × 2 cv F test for comparing supervised classification learning algorithms, Neural Computation, v.11 n.9, p.1885-1892, Nov. 15, 1999 [doi>10.1162/089976699300016007]
• 2: S. Amari , S. Wu, Improving support vector machine classifiers by modifying kernal functions, Neural Networks, v.12 n.6, p.783-789, July 1999 [doi>10.1016/S0893-6080(99)00032-5]
• 3: Francis R. Bach , Gert R. G. Lanckriet , Michael I. Jordan, Multiple kernel learning, conic duality, and the SMO algorithm, Proceedings of the twenty-first international conference on Machine learning, p.6, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015424]
• 4: Ronan Collobert , Samy Bengio , Yoshua Bengio, A parallel mixture of SVMs for very large scale problems, Neural Computation, v.14 n.5, p.1105-1114, May 2002 [doi>10.1162/089976602753633402]
• 5: Robert A. Jacobs , Michael I. Jordan , Steven J. Nowlan , Geoffrey E. Hinton, Adaptive mixtures of local experts, Neural Computation, v.3 n.1, p.79-87, Spring 1991 [doi>10.1162/neco.1991.3.1.79]
• 6: Gert R. G. Lanckriet , Tijl De Bie , Nello Cristianini , Michael I. Jordan , William Stafford Noble, A statistical framework for genomic data fusion, Bioinformatics, v.20 n.16, p.2626-2635, November 2004 [doi>10.1093/bioinformatics/bth294]
• 7: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 8: Lee, W., Verzakov, S., & Duin, R. P. W. (2007). Kernel combination versus classifier combination. Proceedings of the 7th International Workshop on Multiple Classifier Systems (pp. 22--31).
• 9: Darrin P. Lewis , Tony Jebara , William Stafford Noble, Nonstationary kernel combination, Proceedings of the 23rd international conference on Machine learning, p.553-560, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143914]
• 10: Moguerza, J. M., Muñoz, A., & de Diego, I. M. (2004). Improving support vector classification via the combination of multiple sources of information. Proceedings of Structural, Syntactic, and Statistical Pattern Recognition, Joint IAPR International Workshops (pp. 592--600).
• 11: Mosek (2008). The MOSEK optimization tools manual version 5.0 (revision 79). MOSEK ApS, Denmark.
• 12: Paul Pavlidis , Jason Weston , Jinsong Cai , William Noble Grundy, Gene functional classification from heterogeneous data, Proceedings of the fifth annual international conference on Computational biology, p.249-255, April 22-25, 2001, Montreal, Quebec, Canada [doi>10.1145/369133.369228]
• 13: Anders Gorm Pedersen , Henrik Nielsen, Neural Network Prediction of Translation Initiation Sites in Eukaryotes: Perspectives for EST and Genome Analysis, Proceedings of the 5th International Conference on Intelligent Systems for Molecular Biology, p.226-233, June 21-26, 1997
• 14: Alain Rakotomamonjy , Francis Bach , Stéphane Canu , Yves Grandvalet, More efficiency in multiple kernel learning, Proceedings of the 24th international conference on Machine learning, p.775-782, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273594]
• 15: Sören Sonnenburg , Gunnar Rätsch , Christin Schäfer , Bernhard Schölkopf, Large Scale Multiple Kernel Learning, The Journal of Machine Learning Research, 7, p.1531-1565, 12/1/2006

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory