Most likely heteroscedastic Gaussian process regression

ABSTRACT

This paper presents a novel Gaussian process (GP) approach to regression with input-dependent noise rates. We follow Goldberg et al.'s approach and model the noise variance using a second GP in addition to the GP governing the noise-free output value. In contrast to Goldberg et al., however, we do not use a Markov chain Monte Carlo method to approximate the posterior noise variance but a most likely noise approach. The resulting model is easy to implement and can directly be used in combination with various existing extensions of the standard GPs such as sparse approximations. Extensive experiments on both synthetic and real-world data, including a challenging perception problem in robotics, show the effectiveness of most likely heteroscedastic GP regression.

REFERENCES

• 1: Bottou, L., & Bengio, Y. (1995). Convergence properties of the kmeans algorithm. Advances in Neural Information Processing Systems. MIT Press.
• 2: Cawley, G., Talbot, N., & Chapelle, O. (2006). Estimating Predictive Variances with Kernel Ridge Regression. Machine Learning Challenges (pp. 56--77).
• 3: Chan, D., Kohn, R., Nott, D., & Kirby, C. (2006). Locally-adaptive semiparametric estimation of the mean and variance functions in regression models. Journal of Computational and Graphical Statistics.
• 4: Ecker, M., & Heltshe, J. (1994). Geostatistical estimates of scallop abundance. In Case studies in biometry, 107--124. John Wiley and Sons.
• 5: Edakunni, N., Schaal, S., & Vijayakumar, S. (2007). Kernel carpentry for online regression using randomly varying coefficient model. Proc. of IJCAI-07.
• 6: Paul W. Goldberg , Christopher K. I. Williams , Christopher M. Bishop, Regression with input-dependent noise: a Gaussian process treatment, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.493-499, July 1998, Denver, Colorado, United States
• 7: Quoc V. Le , Alex J. Smola , Stéphane Canu, Heteroscedastic Gaussian process regression, Proceedings of the 22nd international conference on Machine learning, p.489-496, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102413]
• 8: David Nott, Semiparametric estimation of mean and variance functions for non-Gaussian data, Computational Statistics, v.21 n.3-4, p.603-620, December 2006 [doi>10.1007/s00180-006-0017-9]
• 9: Opsomer, J., Ruppert, D., Wand, M., Holst, U., & Hossjer, O. (1997). Kriging with nonparametric variance function estimation (Technical Report).
• 10: Carl Edward Rasmussen , Christopher K. I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2005
• 11: Bernhard Schölkopf , Alex J. Smola , Robert C. Williamson , Peter L. Bartlett, New Support Vector Algorithms, Neural Computation, v.12 n.5, p.1207-1245, May 2000 [doi>10.1162/089976600300015565]
• 12: Sigrist, M. (1994). Air Monitoring by Spectroscopic Techniques, vol. 197 of Chemical Analysis Series. John Wiley and Sons.
• 13: Silverman, B. (1985). Some Aspects of the Spline Smoothing Approach to Non-parametric Regression Curve Fitting. Journal of the Royal Statistical Society, 47, 1--52.
• 14: Snelson, E., & Ghahramani, Z. (2006). Variable noise and dimensionality reduction for sparse gaussian processes. Proc. of UAI-06.
• 15: Snelson, E., Rasmussen, C., & Ghahramani, Z. (2003). Warped gaussian processes. NIPS 2003.
• 16: Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. MIT Press.
• 17: C. K. I. Williams, Prediction with Gaussian processes: from linear regression to linear prediction and beyond, Proceedings of the NATO Advanced Study Institute on Learning in graphical models, p.599-621, March 1998, Erice, Italy
• 18: Peter M. Williams, Using neural networks to model conditional multivariate densities, Neural Computation, v.8 n.4, p.843-854, May 1996
• 19: Yuan, M., & Wahba, G. (2004). Doubly penalized likelihood estimator in heteroscedastic regression. Statistics and Probability Letter, 69, 11--20.