MedLDA: maximum margin supervised topic models for regression and classification

ABSTRACT

Supervised topic models utilize document's side information for discovering predictive low dimensional representations of documents; and existing models apply likelihood-based estimation. In this paper, we present a max-margin supervised topic model for both continuous and categorical response variables. Our approach, the maximum entropy discrimination latent Dirichlet allocation (MedLDA), utilizes the max-margin principle to train supervised topic models and estimate predictive topic representations that are arguably more suitable for prediction. We develop efficient variational methods for posterior inference and demonstrate qualitatively and quantitatively the advantages of MedLDA over likelihood-based topic models on movie review and 20 Newsgroups data sets.

REFERENCES

• 1: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 2: Blei, D., & Lafferty, J. (2005). Correlated topic models. Neur. Info. Proc. Sys., 147--154.
• 3: Blei, D., & McAuliffe, J. D. (2007). Supervised topic models. Neur. Info. Proc. Sys., 121--128.
• 4: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 5: Koby Crammer , Yoram Singer, On the algorithmic implementation of multiclass kernel-based vector machines, The Journal of Machine Learning Research, 2, 3/1/2002
• 6: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 7: Lacoste-Jullien, S., Sha, F., & Jordan, M. I. (2008). DiscLDA: Discriminative learning for dimensionality reduction and classification. NIPS, 897--904.
• 8: McCallum, A., Pal, C., Druck, G., & Wang, X. (2006). Multi-conditional learning: generative/discriminative training for clustering and classification. AAAI, 433--439.
• 9: Alex J. Smola , Bernhard Schölkopf, A tutorial on support vector regression, Statistics and Computing, v.14 n.3, p.199-222, August 2004 [doi>10.1023/B:STCO.0000035301.49549.88]
• 10: T. Griffiths, M. S. (2004). Finding scientific topics. Proc. of National Academy of Sci., 5228--5235.
• 11: Teh, Y. W., Newman, D., & Welling, M. (2006). A collapsed variational bayesian inference algorithm for latent dirichlet allocation. NIPS, 1353--1360.
• 12: van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. JMLR, 2579--2605.
• 13: Welling, M., Rosen-Zvi, M., & Hinton, G. (2004). Exponential family harmoniums with an application to information retrieval. NIPS, 1481--1488.
• 14: Jun Zhu , Eric P. Xing , Bo Zhang, Laplace maximum margin Markov networks, Proceedings of the 25th international conference on Machine learning, p.1256-1263, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390314]
• 15: Zhu, J., Xing, E., & Zhang, B. (2008b). Partially observed maximum entropy discrimination Markov networks. Neur. Info. Proc. Sys., 1977--1984.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory