Split variational inference

ABSTRACT

We propose a deterministic method to evaluate the integral of a positive function based on soft-binning functions that smoothly cut the integral into smaller integrals that are easier to approximate. In combination with mean-field approximations for each individual sub-part this leads to a tractable algorithm that alternates between the optimization of the bins and the approximation of the local integrals. We introduce suitable choices for the binning functions such that a standard mean field approximation can be extended to a split mean field approximation without the need for extra derivations. The method can be seen as a revival of the ideas underlying the mixture mean field approach. The latter can be obtained as a special case by taking soft-max functions for the binning.

REFERENCES

• 1: Beal, M. J., & Ghahramani, Z. (2003). The variational Bayesian EM algorithm for incomplete data: with application to scoring graphical model structures. Bayesian Statistics 7 (pp. 453--464). Oxford University Press.
• 2: Christopher M. Bishop , Neil Lawrence , Tommi Jaakkola , Michael I. Jordan, Approximating posterior distributions in belief networks using mixtures, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.416-422, July 1998, Denver, Colorado, United States
• 3: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 4: Brendan J. Frey , David J. C. MacKay, A revolution: belief propagation in graphs with cycles, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.479-485, July 1998, Denver, Colorado, United States
• 5: Jaakkola, T., & Jordan, M. (1996). A variational approach to Bayesian logistic regression problems and their extensions. Proceedings of the Sixth International Workshop on Artificial Intelligence and Statistics.
• 6: Tommi S. Jaakkola , Michael I. Jordon, Improving the mean field approximation via the use of mixture distributions, Learning in graphical models, MIT Press, Cambridge, MA, 1999
• 7: Thomas P. Minka, Expectation Propagation for approximate Bayesian inference, Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, p.362-369, August 02-05, 2001
• 8: Radford M. Neal, Annealed importance sampling, Statistics and Computing, v.11 n.2, p.125-139, April 2001 [doi>10.1023/A:1008923215028]
• 9: Opper, M., & Saad, D. (Eds.). (2001). Advanced mean field methods. MIT Press.
• 10: Parisi, G. (1987). Statistical field theory. Addison-Wesley.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Inference engines

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.1 Interpolation
          Subjects: Smoothing
      G.1.2 Approximation
          Subjects: Special function approximations

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Theory