Compact approximations to Bayesian predictive distributions

ABSTRACT

We provide a general framework for learning precise, compact, and fast representations of the Bayesian predictive distribution for a model. This framework is based on minimizing the KL divergence between the true predictive density and a suitable compact approximation. We consider various methods for doing this, both sampling based approximations, and deterministic approximations such as expectation propagation. These methods are tested on a mixture of Gaussians model for density estimation and on binary linear classification, with both synthetic data sets for visualization and several real data sets. Our results show significant reductions in prediction time and memory footprint.

REFERENCES

• 1: Blake, C., & Merz, C. (1998). UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html.
• 2: Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning. Springer-Verlag. http://www-stat-class.stanford.edu/~tibs/ElemStatLearn/.
• 3: David Maxwell Chickering , David Heckerman, A comparison of scientific and engineering criteria for Bayesian modelselection, Statistics and Computing, v.10 n.1, p.55-62, January 2000 [doi>10.1023/A:1008936501289]
• 4: Ralf Herbrich , Thore Graepel , Colin Campbell, Bayes point machines, The Journal of Machine Learning Research, 1, p.245-279, 9/1/2001 [doi>10.1162/153244301753683717]
• 5: Tommi S. Jaakkola , Michael I. Jordan, Bayesian parameter estimation via variational methods, Statistics and Computing, v.10 n.1, p.25-37, January 2000 [doi>10.1023/A:1008932416310]
• 6: Thomas P. Minka , Rosalind Picard, A family of algorithms for approximate bayesian inference, 2001
• 7: Pál Ruján, Playing billiards in version space, Neural Computation, v.9 n.1, p.99-122, Jan. 1, 1997 [doi>10.1162/neco.1997.9.1.99]
• 8: Tierney, L., & Kadane, J. B. (1986). Accurate approximations for posterior moments and marginal densities. Journal of the American Statistical Association, 81.