Incorporating domain knowledge into topic modeling via Dirichlet Forest priors

ABSTRACT

Users of topic modeling methods often have knowledge about the composition of words that should have high or low probability in various topics. We incorporate such domain knowledge using a novel Dirichlet Forest prior in a Latent Dirichlet Allocation framework. The prior is a mixture of Dirichlet tree distributions with special structures. We present its construction, and inference via collapsed Gibbs sampling. Experiments on synthetic and real datasets demonstrate our model's ability to follow and generalize beyond user-specified domain knowledge.

REFERENCES

• 1: Sugato Basu , Ian Davidson , Kiri Wagstaff, Constrained Clustering: Advances in Algorithms, Theory, and Applications, Chapman & Hall/CRC, 2008
• 2: Blei, D., & Lafferty, J. (2006). Correlated topic models. In Advances in neural information processing systems 18, 147--154. Cambridge, MA: MIT Press.
• 3: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 4: Chaitanya Chemudugunta , America Holloway , Padhraic Smyth , Mark Steyvers, Modeling Documents by Combining Semantic Concepts with Unsupervised Statistical Learning, Proceedings of the 7th International Conference on The Semantic Web, October 26-30, 2008, Karlsruhe, Germany [doi>10.1007/978-3-540-88564-1_15]
• 5: Dennis III, S. Y. (1991). On the hyper-Dirichlet type 1 and hyper-Liouville distributions. Communications in Statistics -- Theory and Methods, 20, 4069--4081.
• 6: Goldberg, A., Fillmore, N., Andrzejewski, D., Xu, Z., Gibson, B., & Zhu, X. (2009). May all your wishes come true: A study of wishes and how to recognize them. Human Language Technologies: Proc. of the Annual Conf. of the North American Chapter of the Assoc. for Computational Linguistics. ACL Press.
• 7: Griffiths, T. L., & Steyvers, M. (2004). Finding scientific topics. Proc. of the Natl. Academy of Sciences of the United States of America, 101, 5228--5235.
• 8: Jerrold R. Griggs , Charles M. Grinstead , David R. Guichard, The number of maximal independent sets in a connected graph, Discrete Mathematics, v.68 n.2-3, p.211-220, Feb, 1988 [doi>10.1016/0012-365X(88)90114-8]
• 9: Wei Li , Andrew McCallum, Pachinko allocation: DAG-structured mixture models of topic correlations, Proceedings of the 23rd international conference on Machine learning, p.577-584, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143917]
• 10: Minka, T. P. (1999). The Dirichlet-tree distribution (Technical Report). http://research.microsoft.com/~minka/papers/dirichlet/minka-dirtree.pdf.
• 11: Tam, Y.-C., & Schultz, T. (2007). Correlated latent semantic model for unsupervised LM adaptation. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (pp. 41--44).
• 12: The Gene Ontology Consortium (2000). Gene Ontology: Tool for the unification of biology. Nature Genetics, 25, 25--29.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Knowledge acquisition

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Experimental design

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Inference engines
      I.2.6 Learning
          Subjects: Language acquisition
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis

General Terms:
Algorithms, Design, Experimentation, Performance, Theory