On the quantitative analysis of deep belief networks

ABSTRACT

Deep Belief Networks (DBN's) are generative models that contain many layers of hidden variables. Efficient greedy algorithms for learning and approximate inference have allowed these models to be applied successfully in many application domains. The main building block of a DBN is a bipartite undirected graphical model called a restricted Boltzmann machine (RBM). Due to the presence of the partition function, model selection, complexity control, and exact maximum likelihood learning in RBM's are intractable. We show that Annealed Importance Sampling (AIS) can be used to efficiently estimate the partition function of an RBM, and we present a novel AIS scheme for comparing RBM's with different architectures. We further show how an AIS estimator, along with approximate inference, can be used to estimate a lower bound on the log-probability that a DBN model with multiple hidden layers assigns to the test data. This is, to our knowledge, the first step towards obtaining quantitative results that would allow us to directly assess the performance of Deep Belief Networks as generative models of data.

REFERENCES

• 1: Bengio, Y., & LeCun, Y. (2007). Scaling learning algorithms towards AI. Large-Scale Kernel Machines. MIT Press.
• 2: Carreira-Perpinan, M., & Hinton, G. (2005). On contrastive divergence learning. 10th Int. Workshop on Artificial Intelligence and Statistics (AISTATS'2005).
• 3: Peter V. Gehler , Alex D. Holub , Max Welling, The rate adapting poisson model for information retrieval and object recognition, Proceedings of the 23rd international conference on Machine learning, p.337-344, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143887]
• 4: Hinton, & Salakhutdinov (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.
• 5: Geoffrey E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Computation, v.14 n.8, p.1771-1800, August 2002 [doi>10.1162/089976602760128018]
• 6: Geoffrey E. Hinton , Simon Osindero , Yee-Whye Teh, A fast learning algorithm for deep belief nets, Neural Computation, v.18 n.7, p.1527-1554, July 2006 [doi>10.1162/neco.2006.18.7.1527]
• 7: Nicolas Le Roux , Yoshua Bengio, Representational power of restricted boltzmann machines and deep belief networks, Neural Computation, v.20 n.6, p.1631-1649, June 2008 [doi>10.1162/neco.2008.04-07-510]
• 8: Neal, R. M. (1993). Probabilistic inference using Markov chain Monte Carlo methods (Technical Report CRG-TR-93-1). Department of Computer Science, University of Toronto.
• 9: Radford M. Neal, Annealed importance sampling, Statistics and Computing, v.11 n.2, p.125-139, April 2001 [doi>10.1023/A:1008923215028]
• 10: Neal, R. M. (2005). Estimating ratios of normalizing constants using linked importance sampling (Technical Report 0511). Department of Statistics, University of Toronto.
• 11: Osindero, S., & Hinton, G. (2008). Modeling image patches with a directed hierarchy of Markov random fields. NIPS 20. Cambridge, MA: MIT Press.
• 12: Ruslan Salakhutdinov , Andriy Mnih , Geoffrey Hinton, Restricted Boltzmann machines for collaborative filtering, Proceedings of the 24th international conference on Machine learning, p.791-798, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273596]
• 13: Skilling, J. (2004). Nested sampling. Bayesian inference and maximum entropy methods in science and engineering, AIP Conference Proceeedings, 735, 395--405.
• 14: Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2006). Modeling human motion using binary latent variables. Advances in Neural Information Processing Systems. MIT Press.
• 15: Wainwright, M. J., Jaakkola, T., & Willsky, A. S. (2005). A new class of upper bounds on the log partition function. IEEE Transactions on Information Theory, 51, 2313--2335.
• 16: Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51, 2282--2312.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Nonmonotonic reasoning and belief revision

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Inference engines; Uncertainty, "fuzzy," and probabilistic reasoning
      I.2.6 Learning
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development

General Terms:
Algorithms, Design, Theory