Archipelago: nonparametric Bayesian semi-supervised learning

ABSTRACT

Semi-supervised learning (SSL), is classification where additional unlabeled data can be used to improve accuracy. Generative approaches are appealing in this situation, as a model of the data's probability density can assist in identifying clusters. Nonparametric Bayesian methods, while ideal in theory due to their principled motivations, have been difficult to apply to SSL in practice. We present a nonparametric Bayesian method that uses Gaussian processes for the generative model, avoiding many of the problems associated with Dirichlet process mixture models. Our model is fully generative and we take advantage of recent advances in Markov chain Monte Carlo algorithms to provide a practical inference method. Our method compares favorably to competing approaches on synthetic and real-world multi-class data.

REFERENCES

• 1: Adams, R. P., Murray, I., & MacKay, D. J. C. (2009). The Gaussian process density sampler. Advances in Neural Information Processing Systems 21 (pp. 9--16).
• 2: Aeberhard, S., Coomans, D., & de Vel, O. (1992). Comparison of classifiers in high dimensional settings (Technical Report 92-02). James Cook University.
• 3: Asuncion, A., & Newman, D. (2007). UCI machine learning repository. http://www.ics.uci.edu/~mlearn/MLRepository.html.
• 4: Beskos, A., Papaspiliopoulos, O., Roberts, G. O., & Fearnhead, P. (2006). Exact and computationally efficient likelihood-based estimation for discretely observed diffusion processes. Journal of the Royal Statistical Society, Series B, 68, 333--382.
• 5: Chu, W., Sindhwani, V., Ghahramani, Z., & Keerthi, S. S. (2007). Relational learning with Gaussian processes. Advances in Neural Information Processing Systems 19 (pp. 289--296).
• 6: Escobar, M. D., & West, M. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90, 577--588.
• 7: Mark Girolami , Simon Rogers, Variational Bayesian multinomial probit regression with Gaussian process priors, Neural Computation, v.18 n.8, p.1790-1817, August 2006 [doi>10.1162/neco.2006.18.8.1790]
• 8: Lawrence, N. D., & Jordan, M. I. (2005). Semi-supervised learning via Gaussian processes. Advances in Neural Information Processing Systems 17 (pp. 753--760).
• 9: Møller, J., Pettitt, A. N., Reeves, R., & Berthelsen, K. K. (2006). An efficient Markov chain Monte Carlo method for distributions with intractable normalising constants. Biometrika, 93, 451--458.
• 10: Murray, I., Ghahramani, Z., & MacKay, D. (2006). MCMC for doubly-intractable distributions. Uncertainty in Artificial Intelligence 22 (pp. 359--366).
• 11: Neal, R. M. (1997). Monte Carlo implementation of Gaussian process models for Bayesian regression and classification (Technical Report 9702). Department of Statistics, University of Toronto.
• 12: Joaquin Quiñonero-Candela , Carl Edward Rasmussen, A Unifying View of Sparse Approximate Gaussian Process Regression, The Journal of Machine Learning Research, 6, p.1939-1959, 12/1/2005
• 13: Carl Edward Rasmussen , Christopher K. I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2005
• 14: Rogers, S., & Girolami, M. (2007). Multi-class semi-supervised learning with the εε-truncated multinomial probit Gaussian process. Journal of Machine Learning Research: Gaussian Processes in Practice (pp. 17--32).
• 15: Sindhwani, V., Chu, W., & Keerthi, S. S. (2007). Semi-supervised Gaussian process classifiers. International Joint Conference on Artificial Intelligence (pp. 1059--1064).
• 16: Teh, Y. W., Seeger, M., & Jordan, M. I. (2005). Semiparametric latent factor models. International Conference on Artificial Intelligence and Statistics 10 (pp. 333--340).
• 17: Christopher K. I. Williams , David Barber, Bayesian Classification With Gaussian Processes, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.12, p.1342-1351, December 1998 [doi>10.1109/34.735807]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Nonparametric statistics; Probabilistic algorithms (including Monte Carlo)

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance