Gaussian process product models for nonparametric nonstationarity

ABSTRACT

Stationarity is often an unrealistic prior assumption for Gaussian process regression. One solution is to predefine an explicit nonstationary covariance function, but such covariance functions can be difficult to specify and require detailed prior knowledge of the nonstationarity. We propose the Gaussian process product model (GPPM) which models data as the pointwise product of two latent Gaussian processes to nonparametrically infer nonstationary variations of amplitude. This approach differs from other nonparametric approaches to covariance function inference in that it operates on the outputs rather than the inputs, resulting in a significant reduction in computational cost and required data for inference. We present an approximate inference scheme using Expectation Propagation. This variational approximation yields convenient GP hyperparameter selection and compact approximate predictive distributions.

REFERENCES

• 1: Gibbs, M. (1997). Bayesian Gaussian processes for regression and classification. Doctoral dissertation, University of Cambridge, Cambridge.
• 2: Paul W. Goldberg , Christopher K. I. Williams , Christopher M. Bishop, Regression with input-dependent noise: a Gaussian process treatment, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.493-499, July 1998, Denver, Colorado, United States
• 3: Simon S. Haykin, Kalman Filtering and Neural Networks, John Wiley & Sons, Inc., New York, NY, 2001
• 4: Higdon, D., Swall, J., & Kern, J. (1999). Nonstationary spatial modeling. In J. Bernardo, J. Berger, A. Dawid and A. Smith (Eds.), Bayesian statistics 6, 761--768. Oxford: Oxford University Press.
• 5: Quoc V. Le , Alex J. Smola , Stéphane Canu, Heteroscedastic Gaussian process regression, Proceedings of the 22nd international conference on Machine learning, p.489-496, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102413]
• 6: Thomas P. Minka , Rosalind Picard, A family of algorithms for approximate bayesian inference, Massachusetts Institute of Technology, 2001
• 7: Nott, D., & Dunsmuir, W. (2002). Estimation of nonstationary spatial covariance structure. Biometrika, 89, 819--829.
• 8: Paciorek, C., & Schervish, M. (2004). Nonstationary covariance functions for Gaussian process regression. Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press.
• 9: Parker, R., & Rice, J. (1985). Discussion of "Some aspects of the spline smoothing approach to nonparametric curve fitting" by B. W. Silverman. Journal of the Royal Statistical Society, Series B, 47, 40--42.
• 10: Pfingsten, T., Kuss, M., & Rasmussen, C. (2006). Nonstationary Gaussian process regression using a latent extension of the input space. www.kyb.mpg.de/publication.html?publ=3985.
• 11: Rasmussen, C. (2000). The infinite Gaussian mixture model. Advances in Neural Information Processing Systems 12 (pp. 554--560). Cambridge, MA: MIT Press.
• 12: Carl Edward Rasmussen , Christopher K. I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2005
• 13: Sampson, P., & Guttorp, P. (1992). Nonparametric estimation of nonstationary spatial covariance structure. Journal of the American Statistical Association, 87, 108--119.
• 14: Schmidt, A. M., & O'Hagan, A. (2003). Bayesian inference for nonstationary spatial covariance structure via spatial deformations. Journal of the Royal Statistical Society, Series B, 65, 745--758.
• 15: Seeger, M. (2005). Expectation propagation for exponential families (Technical Report). Technical report, University of California at Berkeley, 2005. See www.kyb.tuebingen.mpg.de/bs/people/seeger.
• 16: Snelson, E., & Ghahramani, Z. (2006). Variable noise and dimensionality reduction for sparse Gaussian processes. Proceedings of the 22nd International Conference on Uncertainty in Artificial Intelligence.
• 17: Turner, R., & Sahani, M. (2008). Modeling natural sounds with modulation cascade processes. Advances in Neural Information Processing Systems 21. Cambridge, MA: MIT Press.
• 18: Zoeter, O., & Heskes, T. (2005). Gaussian quadrature based expectation propagation. Proceedings of Artificial Intelligence and Statistics 2005.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Nonparametric statistics

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.2 Approximation

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Inference engines
      I.2.6 Learning
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Theory