Combining concept hierarchies and statistical topic models

ABSTRACT

Statistical topic models provide a general data-driven framework for automated discovery of high-level knowledge from large collections of text documents. While topic models can potentially discover a broad range of themes in a data set, the interpretability of the learned topics is not always ideal. Human-defined concepts, on the other hand, tend to be semantically richer due to careful selection of words to define concepts but they tend not to cover the themes in a data set exhaustively. In this paper, we propose a probabilistic framework to combine a hierarchy of human-defined semantic concepts with statistical topic models to seek the best of both worlds. Experimental results using two different sources of concept hierarchies and two collections of text documents indicate that this combination leads to systematic improvements in the quality of the associated language models as well as enabling new techniques for inferring and visualizing the semantics of a document.

REFERENCES

• 1: D. M. Blei and J. D. Lafferty. Correlated topic models. In NIPS, 2005.
• 2: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 3: D. Boyd-Graber, D. Blei, and X. Zhu. A topic model for word sense disambiguation. In Proc. 2007 Joint Conf. Empirical Methods in Nat'l. Lang. Processing and Compt'l. Nat'l. Lang. Learning, pages 1024--1033, 2007.
• 4: C. Chemudugunta, A. Holloway, P. Smyth, and M. Steyvers. Modeling documents by combining semantic concepts with unsupervised statistical learning. In International Semantic Web Conference, 2008.
• 5: C. Chemudugunta, P. Smyth, and M. Steyvers. Modeling general and specific aspects of documents with a probabilistic topic model. In NIPS 19, 2007.
• 6: C. Chemudugunta, P. Smyth, and M. Steyvers. Text modeling using unsupervised topic models and concept hierarchies. url: http://arxiv.org/abs/0808.0973. August 2008.
• 7: T. L. Griffiths and M. Steyvers. Finding scientific topics. In Proc. of Nat'l. Academy of Science, volume 101, pages 5228--5235, 2004.
• 8: G. Ifrim, M. Theobald, and G. Weikum. Learning word-to-concept mappingsfor automatic text classification. In 22nd ICML-LWS, pages 18--26, 2005.
• 9: David Mimno , Wei Li , Andrew McCallum, Mixtures of hierarchical topics with Pachinko allocation, Proceedings of the 24th international conference on Machine learning, p.633-640, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273576]
• 10: Mark Steyvers , Padhraic Smyth , Michal Rosen-Zvi , Thomas Griffiths, Probabilistic author-topic models for information discovery, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014087]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
ontologies, semantic concepts, statistical topic models, unsupervised learning