Independent factor topic models

ABSTRACT

Topic models such as Latent Dirichlet Allocation (LDA) and Correlated Topic Model (CTM) have recently emerged as powerful statistical tools for text document modeling. In this paper, we improve upon CTM and propose Independent Factor Topic Models (IFTM) which use linear latent variable models to uncover the hidden sources of correlation between topics. There are 2 main contributions of this work. First, by using a sparse source prior model, we can directly visualize sparse patterns of topic correlations. Secondly, the conditional independence assumption implied in the use of latent source variables allows the objective function to factorize, leading to a fast Newton-Raphson based variational inference algorithm. Experimental results on synthetic and real data show that IFTM runs on average 3--5 times faster than CTM, while giving competitive performance as measured by perplexity and loglikelihood of held-out data.

REFERENCES

• 1: Attias, H. (2000). A variational bayesian framework for graphical models. Advances in Neural Information Processing Systems (NIPS) (pp. 209--215).
• 2: Blei, D. M., Griffiths, T., Jordan, M. I., & Tenenbaum, J. (2004). Hierarchical topic models and the nested chinese restaurant process. Advances in Neural Information Processing Systems (NIPS) (pp. 17--24).
• 3: Blei, D. M., & Lafferty, J. D. (2006). Correlated topic models. Advances in Neural Information Processing Systems (NIPS) (pp. 147--154).
• 4: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 5: Everitt, B. S. (1984). An introduction to latent variable models. London: Chapman and Hall.
• 6: Mark Girolami, A Variational Method for Learning Sparse and Overcomplete Representations, Neural Computation, v.13 n.11, p.2517-2532, November 2001 [doi>10.1162/089976601753196003]
• 7: Griffiths, T., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Sciences (pp. 5228--5235).
• 8: Tommi S. Jaakkola , Michael I. Jordan, Variational methods for inference and estimation in graphical models, Massachusetts Institute of Technology, 1997
• 9: Joreskog, K. G. (1967). Some contributions to maximum likelihood factor analysis. Psychometrika, 32, 443--482.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.5 Document Capture
          Subjects: Document analysis

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing; Experimental design

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Abstracting methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis

General Terms:
Algorithms, Design, Performance, Theory