Hierarchic Bayesian models for kernel learning

ABSTRACT

The integration of diverse forms of informative data by learning an optimal combination of base kernels in classification or regression problems can provide enhanced performance when compared to that obtained from any single data source. We present a Bayesian hierarchical model which enables kernel learning and present effective variational Bayes estimators for regression and classification. Illustrative experiments demonstrate the utility of the proposed method. Matlab code replicating results reported is available at http://www.dcs.gla.ac.uk/~srogers/kernel_comb.html.

REFERENCES

• 1: Andrews, D., & Mallows, C. (1974). Scale mixtures of Normal distributions. Journal of the Royal Statistical Society, Series B, 36, 99--102.
• 2: Francis R. Bach , Gert R. G. Lanckriet , Michael I. Jordan, Multiple kernel learning, conic duality, and the SMO algorithm, Proceedings of the twenty-first international conference on Machine learning, p.6, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015424]
• 3: Bach, F. R., Thibaux, R., & Jordan, M. I. (2005). Computing regularization paths for learning multiple kernels. In L. K. Saul. Y. Weiss and L. Bottou (Eds.), Advances in neural information processing systems 17. Cambridge. MA: MIT Press.
• 4: Beal, M. (2003). Variational algorithms for approximate bayesian inference. Doctoral dissertation, University College London.
• 5: Christopher M. Bishop , Michael E. Tipping, Variational Relevance Vector Machines, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.46-53, June 30-July 03, 2000
• 6: Bousquet, O., & Herrmann, D. J. L. (2003). On the complexity of learning the kernel matrix. In S. T. S. Becker and K. Obermayer (Eds.), Advances in neural information processing systems 15, 399--406. Cambridge, MA: MIT Press.
• 7: Crammer, K., Keshet, J., & Singer, Y. (2003). Kernel design using boosting. In S. T. S. Becker and K. Obermayer (Eds.), Advances in neural information processing systems 15, 537--544. Cambridge, MA: MIT Press.
• 8: Cristianini, N., Shawe-Taylor, J., Elisseeff, A., & Kandola. J. (2002). On kernel-target alignment. Advances in Neural Information Processing Systems 14. Cambridge, MA: MIT Press.
• 9: Glenn Fung , Murat Dundar , Jinbo Bi , Bharat Rao, A fast iterative algorithm for fisher discriminant using heterogeneous kernels, Proceedings of the twenty-first international conference on Machine learning, p.40, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015409]
• 10: S. R. Gunn , J. S. Kandola, Structural Modelling with Sparse Kernels, Machine Learning, v.48 n.1-3, p.137-163, 2002 [doi>10.1023/A:1013903804720]
• 11: Tommi S. Jaakkola , Michael I. Jordan, Variational methods for inference and estimation in graphical models, 1997
• 12: Michael I. Jordan , Zoubin Ghahramani , Tommi S. Jaakkola , Lawrence K. Saul, An Introduction to Variational Methods for Graphical Models, Machine Learning, v.37 n.2, p.183-233, Nov.1.1999 [doi>10.1023/A:1007665907178]
• 13: Kolenda, T., Hansen, L., Larsen, J., & Winther, O. (2002). Independent component analysis for understanding multimedia content. Proceedings of IEEE Workshop on Neural Networks for Signal Processing XII (pp. 757--766).
• 14: John Lafferty , Guy Lebanon, Diffusion Kernels on Statistical Manifolds, The Journal of Machine Learning Research, 6, p.129-163, 9/1/2005
• 15: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 16: Neil D. Lawrence , Marta Milo , Mahesan Niranjan , Penny Rashbass , Stephan Soullier, Reducing the variability in cDNA microarray image processing by Bayesian inference, Bioinformatics, v.20 n.4, p.518-526, March 2004 [doi>10.1093/bioinformatics/btg438]
• 17: David J. C. MacKay, Information Theory, Inference & Learning Algorithms, Cambridge University Press, New York, NY, 2002
• 18: Thomas P. Minka , Rosalind Picard, A family of algorithms for approximate bayesian inference, 2001
• 19: Ong, C. S., Smola, A. J., & Williamson, R. C. (2003). Hyperkernels. In S. T. S. Becker and K. Obermayer (Eds.), Advances in neural information processing systems 15, 478--485. Cambridge, MA: MIT Press.
• 20: John Shawe-Taylor , Nello Cristianini, Kernel Methods for Pattern Analysis, Cambridge University Press, New York, NY, 2004
• 21: Michael E. Tipping, Sparse bayesian learning and the relevance vector machine, The Journal of Machine Learning Research, 1, p.211-244, 9/1/2001
• 22: Tsang, I. W., & Kwok, J. T. (2004). Efficient hyperkernel learning using second-order cone programming. Proceedings of the 15th European Conference on Machine Learning (pp. 453--464).
• 23: Zhihua Zhang , Dit-Yan Yeung , James T. Kwok, Bayesian inference for transductive learning of kernel matrix using the Tanner-Wong data augmentation algorithm, Proceedings of the twenty-first international conference on Machine learning, p.118, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015368]