Efficient methods for topic model inference on streaming document collections

ABSTRACT

Topic models provide a powerful tool for analyzing large text collections by representing high dimensional data in a low dimensional subspace. Fitting a topic model given a set of training documents requires approximate inference techniques that are computationally expensive. With today's large-scale, constantly expanding document collections, it is useful to be able to infer topic distributions for new documents without retraining the model. In this paper, we empirically evaluate the performance of several methods for topic inference in previously unseen documents, including methods based on Gibbs sampling, variational inference, and a new method inspired by text classification. The classification-based inference method produces results similar to iterative inference methods, but requires only a single matrix multiplication. In addition to these inference methods, we present SparseLDA, an algorithm and data structure for evaluating Gibbs sampling distributions. Empirical results indicate that SparseLDA can be approximately 20 times faster than traditional LDA and provide twice the speedup of previously published fast sampling methods, while also using substantially less memory.

REFERENCES

• 1: A. Banerjee and S. Basu. Topic models over text streams: A study of batch and online unsupervised learning. In SIAM-DM, 2007.
• 2: Adam L. Berger , Vincent J. Della Pietra , Stephen A. Della Pietra, A maximum entropy approach to natural language processing, Computational Linguistics, v.22 n.1, p.39-71, March 1996
• 3: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 4: T. Griffiths and M. Steyvers. Finding scientific topics. PNAS, 101(Suppl. 1):5228--5235, 2004.
• 5: A. K. McCallum. MALLET: A machine learning for language toolkit. http://mallet.cs.umass.edu, 2002.
• 6: D. Newman, A. Asuncion, P. Smyth, and M. Welling. Distributed inference for latent dirichlet allocation. In NIPS, 2007.
• 7: Ian Porteous , David Newman , Alexander Ihler , Arthur Asuncion , Padhraic Smyth , Max Welling, Fast collapsed gibbs sampling for latent dirichlet allocation, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA [doi>10.1145/1401890.1401960]
• 8: Xiaodan Song , Ching-Yung Lin , Belle L. Tseng , Ming-Ting Sun, Modeling and predicting personal information dissemination behavior, Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081925]
• 9: Y. W. Teh, D. Newman, and M. Welling. A collapsed variational bayesian inference algorithm for latent dirichlet allocation. In NIPS, 2006.
• 10: X. Wei, J. Sun, and X. Wang. Dynamic mixture models for multiple time series. In IJCAI, 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

General Terms:
Design, Experimentation, Performance

Keywords:
inference, topic modeling