Learning the unified kernel machines for classification

ABSTRACT

Kernel machines have been shown as the state-of-the-art learning techniques for classification. In this paper, we propose a novel general framework of learning the Unified Kernel Machines (UKM) from both labeled and unlabeled data. Our proposed framework integrates supervised learning, semi-supervised kernel learning, and active learning in a unified solution. In the suggested framework, we particularly focus our attention on designing a new semi-supervised kernel learning method, i.e., Spectral Kernel Learning (SKL), which is built on the principles of kernel target alignment and unsupervised kernel design. Our algorithm is related to an equivalent quadratic programming problem that can be efficiently solved. Empirical results have shown that our method is more effective and robust to learn the semi-supervised kernels than traditional approaches. Based on the framework, we present a specific paradigm of unified kernel machines with respect to Kernel Logistic Regresions (KLR), i.e., Unified Kernel Logistic Regression (UKLR). We evaluate our proposed UKLR classification scheme in comparison with traditional solutions. The promising results show that our proposed UKLR paradigm is more effective than the traditional classification approaches.

REFERENCES

• 1: M. Belkin and I. M. and P. Niyogi. Regularization and semi-supervised learning on large graphs. In COLT, 2004.
• 2: Mikhail Belkin , Partha Niyogi, Semi-Supervised Learning on Riemannian Manifolds, Machine Learning, v.56 n.1-3, p.209-239 [doi>10.1023/B:MACH.0000033120.25363.1e]
• 3: E. Chang, S. C. Hoi, X. Wang, W.-Y. Ma, and M. Lyu. A unified machine learning framework for large-scale personalized information management. In The 5th Emerging Information Technology Conference, NTU Taipei, 2005.
• 4: E. Chang and M. Lyu. Unified learning paradigm for web-scale mining. In Snowbird Machine Learning Workshop, 2006.
• 5: O. Chapelle, A. Zien, and B. Scholkopf. Semi-supervised learning. MIT Press, 2006.
• 6: F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
• 7: D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. In NIPS, volume 7, pages 705--712, 1995.
• 8: N. Cristianini, J. Shawe-Taylor, and A. Elisseeff. On kernel-target alignment. JMLR, 2002.
• 9: Shai Fine , Ran Gilad-Bachrach , Eli Shamir, Query by committee, linear separation and random walks, Theoretical Computer Science, v.284 n.1, p.25-51, 6 July 2002 [doi>10.1016/S0304-3975(01)00079-2]
• 10: Yoav Freund , H. Sebastian Seung , Eli Shamir , Naftali Tishby, Selective Sampling Using the Query by Committee Algorithm, Machine Learning, v.28 n.2-3, p.133-168, Aug./Sept. 1997 [doi>10.1023/A:1007330508534]
• 11: Steven C. H. Hoi , Rong Jin , Michael R. Lyu, Large-scale text categorization by batch mode active learning, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135870]
• 12: S. B. C. M. J. A. K. Suykens, G. Horvath and J. Vandewalle. Advances in Learning Theory: Methods, Models and Applications. NATO Science Series: Computer & Systems Sciences, 2003.
• 13: R. Kondor and J. Lafferty. Diffusion kernels on graphs and other discrete structures. 2002.
• 14: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 15: G. Lanckriet, L. Ghaoui, C. Bhattacharyya, and M. Jordan. Minimax probability machine. In Advances in Neural Information Processing Systems 14, 2002.
• 16: R. Liere and P. Tadepalli. Active learning with committees for text categorization. In Proceedings 14th Conference of the American Association for Artificial Intelligence (AAAI), pages 591--596, MIT Press, 1997.
• 17: Ron Meir , Gunnar Rätsch, An introduction to boosting and leveraging, Advanced lectures on machine learning, Springer-Verlag New York, Inc., New York, NY, 2003
• 18: A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. In In Advances in Neural Information Processing Systems 14, 2001.
• 19: Nicholas Roy , Andrew McCallum, Toward Optimal Active Learning through Sampling Estimation of Error Reduction, Proceedings of the Eighteenth International Conference on Machine Learning, p.441-448, June 28-July 01, 2001
• 20: Bernhard Schölkopf , Alexander Smola , Klaus-Robert Müller, Nonlinear component analysis as a kernel eigenvalue problem, Neural Computation, v.10 n.5, p.1299-1319, July 1, 1998 [doi>10.1162/089976698300017467]
• 21: A. Smola and R. Kondor. Kernels and regularization on graphs. In Intl. Conf. on Learning Theory, 2003.
• 22: M. Szummer and T. Jaakkola. Partially labeled classification with markov random walks. In Advances in Neural Information Processing Systems, 2001.
• 23: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 24: Simon Tong , Daphne Koller, Support Vector Machine Active Learning with Application sto Text Classification, Proceedings of the Seventeenth International Conference on Machine Learning, p.999-1006, June 29-July 02, 2000
• 25: V. N. Vapnik. Statistical Learning Theory. John Wiley & Sons, 1998.
• 26: G. Wu, Z. Zhang, and E. Y. Chang. Kronecker factorization for speeding up kernel machines. In SIAM Int. Conference on Data Mining (SDM), 2005.
• 27: T. Zhang and R. K. Ando. Analysis of spectral kernel design based semi-supervised learning. In NIPS, 2005.
• 28: D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schlkopf. Learning with local and global consistency. In NIPS'16, 2005.
• 29: J. Zhu and T. Hastie. Kernel logistic regression and the import vector machine. In NIPS 14, pages 1081--1088, 2001.
• 30: X. Zhu. Semi-supervised learning literature survey. Technical report, Computer Sciences TR 1530, University of Wisconsin - Madison, 2005.
• 31: X. Zhu, Z. Ghahramani, and J. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proc. ICML' 2003, 2003.
• 32: X. Zhu, J. Kandola, Z. Ghahramani, and J. Lafferty. Nonparametric transforms of graph kernels for semi-supervised learning. In NIPS 2005, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms, Design, Experimentation

Keywords:
active learning, classification, kernel logistic regressions, kernel machines, semi-supervised learning, spectral kernel learning, supervised learning, unsupervised kernel design