The Bayesian backfitting relevance vector machine

ABSTRACT

Traditional non-parametric statistical learning techniques are often computationally attractive, but lack the same generalization and model selection abilities as state-of-the-art Bayesian algorithms which, however, are usually computationally prohibitive. This paper makes several important contributions that allow Bayesian learning to scale to more complex, real-world learning scenarios. Firstly, we show that backfitting --- a traditional non-parametric, yet highly efficient regression tool --- can be derived in a novel formulation within an expectation maximization (EM) framework and thus can finally be given a probabilistic interpretation. Secondly, we show that the general framework of sparse Bayesian learning and in particular the relevance vector machine (RVM), can be derived as a highly efficient algorithm using a Bayesian version of backfitting at its core. As we demonstrate on several regression and classification benchmarks, Bayesian backfitting offers a compelling alternative to current regression methods, especially when the size and dimensionality of the data challenge computational resources.

REFERENCES

• 1: Christopher M. Bishop , Michael E. Tipping, Variational Relevance Vector Machines, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.46-53, June 30-July 03, 2000
• 2: Csató, L., & Opper, M. (2001). Sparse representation for Gaussian process models. In (Leen et al., 2001), 444--450.
• 3: Kenji Fukumizu , Francis R. Bach , Michael I. Jordan, Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces, The Journal of Machine Learning Research, 5, p.73-99, 12/1/2004
• 4: Ghahramani, Z., & Beal, M. J. (2000). Variational inference for Bayesian mixtures of factor analysers. Advances in Neural Information Processing Systems 12 (pp. 509--514). Cambridge, MA: MIT Press.
• 5: Hastie, T. J., & Tibshirani, R. J. (1990). Generalized additive models. No. 43 in Monographs on Statistics and Applied Probability. Chapman & Hall.
• 6: Tommi S. Jaakkola , Michael I. Jordan, Bayesian parameter estimation via variational methods, Statistics and Computing, v.10 n.1, p.25-37, January 2000 [doi>10.1023/A:1008932416310]
• 7: Leen, T. K., Diettrich, T. G., & Tresp, V. (Eds.). (2001). Advances in neural information processing systems 13, vol. 13. Cambridge, MA: MIT Press.
• 8: David J. C. MacKay, Comparison of approximate methods for handling hyperparameters, Neural Computation, v.11 n.5, p.1035-1068, July 1, 1999 [doi>10.1162/089976699300016331]
• 9: Massey, W. F. (1965). Principal component regression in exploratory statistical research. Journal of the American Statistical Association, 60, 234--246.
• 10: Radford M. Neal, Bayesian leaning for neural networks, University of Toronto, Toronto, Ont., Canada, 1996
• 11: William H. Press , Saul A. Teukolsky , William T. Vetterling , Brian P. Flannery, Numerical recipes in C (2nd ed.): the art of scientific computing, Cambridge University Press, New York, NY, 1992
• 12: Stefan Schaal , Sethu Vijayakumar , Christopher G. Atkeson, Local dimensionality reduction, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.633-639, July 1998, Denver, Colorado, United States
• 13: Michael E. Tipping, Sparse bayesian learning and the relevance vector machine, The Journal of Machine Learning Research, 1, p.211-244, 9/1/2001
• 14: Tipping, M. E., & Faul, A. C. (2003). Fast marginal likelihood maximization for sparse Bayesian models. Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics.
• 15: Sethu Vijayakumar , Stefan Schaal, Locally Weighted Projection Regression: Incremental Real Time Learning in High Dimensional Space, Proceedings of the Seventeenth International Conference on Machine Learning, p.1079-1086, June 29-July 02, 2000
• 16: Williams, C. K. I., & Rasmussen, C. E. (1996). Gaussian processes for regression. Advances in Neural Information Processing Systems 8 (pp. 514--520). Cambridge, MA: MIT Press.
• 17: Williams, C. K. I., & Seeger, M. (2001). Using the Nyströöm method to speed up kernel machines. In (Leen et al., 2001), 682--688.
• 18: Wold, H. (1975). Soft modeling by latent variables: The nonlinear iterative partial least squares approach. In J. Gani (Ed.), Perspectives in probability and statistics, papers in honour of M. S. Bartlett, 520--540. London: Academic Press.