Local distance preservation in the GP-LVM through back constraints

ABSTRACT

The Gaussian process latent variable model (GP-LVM) is a generative approach to nonlinear low dimensional embedding, that provides a smooth probabilistic mapping from latent to data space. It is also a non-linear generalization of probabilistic PCA (PPCA) (Tipping & Bishop, 1999). While most approaches to non-linear dimensionality methods focus on preserving local distances in data space, the GP-LVM focusses on exactly the opposite. Being a smooth mapping from latent to data space, it focusses on keeping things apart in latent space that are far apart in data space. In this paper we first provide an overview of dimensionality reduction techniques, placing the emphasis on the kind of distance relation preserved. We then show how the GP-LVM can be generalized, through back constraints, to additionally preserve local distances. We give illustrative experiments on common data sets.

REFERENCES

• 1: Bilmes, J., Malkin, J., Li, X., Harada, S., Kilanski, K., Kirchhoff, K., Wright, R., Subramanya, A., Landay, J., Dowden, P., & Chizeck, H. (2006). The vocal joystick. Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing. IEEE. To appear.
• 2: Christopher M. Bishop , Markus Svensén , Christopher K. I. Williams, GTM: the generative topographic mapping, Neural Computation, v.10 n.1, p.215-234, Jan. 1, 1998 [doi>10.1162/089976698300017953]
• 3: Keith Grochow , Steven L. Martin , Aaron Hertzmann , Zoran Popović, Style-based inverse kinematics, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 4: Hinton, G., & Roweis, S. (2003). Stochastic neighbor embedding. Advances in Neural Information Processing Systems 15. Cambridge, MA: MIT Press.
• 5: Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science, 268, 1158--1161.
• 6: Jolliffe, I. T. (1986). Principal component analysis. New York: Springer-Verlag.
• 7: Lawrence, N. D. (2004). Gaussian process models for visualisation of high dimensional data. Advances in Neural Information Processing Systems (pp. 329--336). Cambridge, MA: MIT Press.
• 8: Lawrence, N. D. (2005). Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research, 6, 1783--1816.
• 9: Lowe, D., & Tipping, M. E. (1996). Feed-forward neural networks and topographic mappings for exploratory data analysis. Neural Computing and Applications, 4.
• 10: MacKay, D. J. C. (1995). Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research, A, 354, 73--80.
• 11: Mardia, K. V., Kent, J. T., & Bibby, J. M. (1979). Multivariate analysis. London: Academic Press.
• 12: Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290, 2323--2326.
• 13: Sammon, J. W. (1969). A nonlinear mapping for data structure analysis. IEEE Transactions on Computers, C-18, 401--409.
• 14: Bernhard Schölkopf , Alexander Smola , Klaus-Robert Müller, Nonlinear component analysis as a kernel eigenvalue problem, Neural Computation, v.10 n.5, p.1299-1319, July 1, 1998 [doi>10.1162/089976698300017467]
• 15: Shon, A. P., Grochow, K., Hertzmann, A., & Rao, R. P. N. (2006). Learning shared latent structure for image synthesis and robotic imitation. In (Weiss et al., 2006).
• 16: Tenenbaum, J. B., Silva, V. d., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319--2323.
• 17: Tipping, M. E., & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society, B, 6, 611--622.
• 18: Raquel Urtasun , David J. Fleet , Aaron Hertzmann , Pascal Fua, Priors for People Tracking from Small Training Sets, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.403-410, October 17-20, 2005 [doi>10.1109/ICCV.2005.193]
• 19: Wang, J. M., Fleet, D. J., & Hertzmann, A. (2006). Gaussian process dynamical models. In (Weiss et al., 2006).
• 20: Kilian Q. Weinberger , Fei Sha , Lawrence K. Saul, Learning a kernel matrix for nonlinear dimensionality reduction, Proceedings of the twenty-first international conference on Machine learning, p.106, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015345]
• 21: Weiss, Y., Schölkopf, B., & Platt, J. C. (Eds.). (2006). Advances in neural information processing systems, vol. 18. Cambridge, MA: MIT Press.
• 22: Alexander Zien , Joaquin Quiñonero Candela, Large margin non-linear embedding, Proceedings of the 22nd international conference on Machine learning, p.1060-1067, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102485]

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo); Stochastic processes

General Terms:
Algorithms, Theory