Herding dynamical weights to learn

ABSTRACT

A new "herding" algorithm is proposed which directly converts observed moments into a sequence of pseudo-samples. The pseudo-samples respect the moment constraints and may be used to estimate (unobserved) quantities of interest. The procedure allows us to sidestep the usual approach of first learning a joint model (which is intractable) and then sampling from that model (which can easily get stuck in a local mode). Moreover, the algorithm is fully deterministic, avoiding random number generation) and does not need expensive operations such as exponentiation.

REFERENCES

• 1: Besag, J. (1977). Efficiency of pseudo-likelihood estimation for simple Gaussian fields. Biometrika, 64, 616--618.
• 2: Ganapathi, V., Vickrey, D., Duchi, J., & Koller, D. (2008). Constrained approximate maximum entropy learning. Proceedings of the Twenty-fourth Conference on Uncertainty in AI (pp. 196--203).
• 3: Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721--741.
• 4: Geoffrey E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Computation, v.14 n.8, p.1771-1800, August 2002 [doi>10.1162/089976602760128018]
• 5: Hopfield, J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79, 2554--2558.
• 6: Aapo Hyvärinen, Estimation of Non-Normalized Statistical Models by Score Matching, The Journal of Machine Learning Research, 6, p.695-709, 12/1/2005
• 7: Jaynes, E. (1957). Information theory and statistical mechanics. Physical Review, 106, 620--630.
• 8: John Lafferty, Additive models, boosting, and inference for generalized divergences, Proceedings of the twelfth annual conference on Computational learning theory, p.125-133, July 07-09, 1999, Santa Cruz, California, United States [doi>10.1145/307400.307422]
• 9: Lebanon, G., & Lafferty, J. (2002). Boosting and maximum likelihood for exponential models. Neural Information Processing Systems (pp. 447--454).
• 10: Levina, A., Herrmann, J., & Geisel, T. (2007). Dynamical synapses causing self-organized criticality in neural networks. Nature Physics, 3, 857--860.
• 11: Wolfgang Maass , Anthony M. Zador, Dynamic stochastic synapses as computational units, Neural Computation, v.11 n.4, p.903-917, May 15, 1999 [doi>10.1162/089976699300016494]
• 12: Iain Murray , Zoubin Ghahramani, Bayesian learning in undirected graphical models: approximate MCMC algorithms, Proceedings of the 20th conference on Uncertainty in artificial intelligence, p.392-399, July 07-11, 2004, Banff, Canada
• 13: Radford M. Neal, Connectionist learning of belief networks, Artificial Intelligence, v.56 n.1, p.71-113, July 1992 [doi>10.1016/0004-3702(92)90065-6]
• 14: Lovorka Pantic , Joaquín J. Torres , Hilbert J. Kappen , Stan C. A. M. Gielen, Associative memory with dynamic synapses, Neural Computation, v.14 n.12, p.2903-2923, December 2002 [doi>10.1162/089976602760805331]
• 15: Parise, S., & Welling, M. (2005). Learning in markov random fields: An empirical study. Proc. of the Joint Statistical Meeting.
• 16: Teh, Y., & Welling, M. (2002). The unified propagation and scaling algorithm. Neural Information Processing Systems (pp. 953--960).
• 17: Tijmen Tieleman, Training restricted Boltzmann machines using approximations to the likelihood gradient, Proceedings of the 25th international conference on Machine learning, p.1064-1071, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390290]
• 18: Welling, M., & Parise, S. (2006). Bayesian random fields: The Bethe-Laplace approximation. Proc. of the Conf. on Uncertainty in Artificial Intelligence (pp. 512--519).
• 19: Younes, L. (1999). On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics An International Journal of Probability and Stochastic Processes, 65, 177--228.
• 20: Yuille, A. (2004). The convergence of contrastive divergences. Advances in Neural Information Processing Systems (pp. 1593--1600).
• 21: Song Chun Zhu , Xiuwen Liu, Learning in Gibbsian Fields: How Accurate and How Fast Can It Be?, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.7, p.1001-1006, July 2002 [doi>10.1109/TPAMI.2002.1017626]
• 22: Song Chun Zhu , Ying Nian Wu , David Mumford, Minimax entropy principle and its application to texture modeling, Neural Computation, v.9 n.9, p.1627-1660, Nov. 15, 1997 [doi>10.1162/neco.1997.9.8.1627]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo)

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory