Evaluation methods for topic models

ABSTRACT

A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.

REFERENCES

• 1: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 2: Chib, S. (1995). Marginal likelihood from the Gibbs output. J. American Stat. Assoc., 90, 1313--1321.
• 3: Del Moral, P., Doucet, A., & Jasra, A. (2006). Sequential Monte Carlo samplers. J. Royal Stat. Soc. B, 68, 1--26.
• 4: Gogate, V., Bidyuk, B., & Dechter, R. (2007). Studies in lower bounding probability of evidence using the Markov inequality. Proc. Conf. on Uncertainty in Artificial Intelligence (pp. 141--148).
• 5: Griffiths, T., & Steyvers, M. (2004). Finding scientific topics. Proc. Natl. Acad. Sci., 101, 5228--5235.
• 6: Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenen-baum, J. B. (2005). Integrating topics and syntax. Proc. Neural Information Processing Systems (pp. 537--544).
• 7: Wei Li , Andrew McCallum, Pachinko allocation: DAG-structured mixture models of topic correlations, Proceedings of the 23rd international conference on Machine learning, p.577-584, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143917]
• 8: McCallum, A. (2002). MALLET: A machine learning for language toolkit. http://mallet.cs.umass.edu.
• 9: Murray, I., & Salakhutdinov, R. (2009). Evaluating probabilities under high-dimensional latent variable models. Proc. Neural Information Processing Systems (pp. 1137--1144).
• 10: Radford M. Neal, Annealed importance sampling, Statistics and Computing, v.11 n.2, p.125-139, April 2001 [doi>10.1023/A:1008923215028]
• 11: Newton, M. A., & Raftery, A. E. (1994). Approximate Bayesian inference with the weighted likelihood bootstrap. J. Royal Stat. Soc. B, 56, 3--48.
• 12: Michal Rosen-Zvi , Thomas Griffiths , Mark Steyvers , Padhraic Smyth, The author-topic model for authors and documents, Proceedings of the 20th conference on Uncertainty in artificial intelligence, p.487-494, July 07-11, 2004, Banff, Canada
• 13: Teh, Y., Jordan, M., Beal, M., & Blei, D. (2006). Hierarchical Dirichlet processes. J. American Stat. Assoc., 101, 1566--1581.
• 14: Hanna M. Wallach, Topic modeling: beyond bag-of-words, Proceedings of the 23rd international conference on Machine learning, p.977-984, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143967]
• 15: Wallach, H. M. (2008). Structured topic models for language. PhD thesis, University of Cambridge.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing; Experimental design; Probabilistic algorithms (including Monte Carlo)

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Experimentation, Performance, Theory