Probabilistic author-topic models for information discovery

ABSTRACT

We propose a new unsupervised learning technique for extracting information from large text collections. We model documents as if they were generated by a two-stage stochastic process. Each author is represented by a probability distribution over topics, and each topic is represented as a probability distribution over words for that topic. The words in a multi-author paper are assumed to be the result of a mixture of each authors' topic mixture. The topic-word and author-topic distributions are learned from data in an unsupervised manner using a Markov chain Monte Carlo algorithm. We apply the methodology to a large corpus of 160,000 abstracts and 85,000 authors from the well-known CiteSeer digital library, and learn a model with 300 topics. We discuss in detail the interpretation of the results discovered by the system including specific topic and author models, ranking of authors by topic and topics by author, significant trends in the computer science literature between 1990 and 2002, parsing of abstracts by topics and authors and detection of unusual papers by specific authors. An online query interface to the model is also discussed that allows interactive exploration of author-topic models for corpora such as CiteSeer.

REFERENCES

• 1: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003
• 2: Buntine, W.L. (1994) Operations for learning with graphical models, Journal of Artificial Intelligence Research 2, pp. 159--225.
• 3: Douglass R. Cutting , David R. Karger , Jan O. Pedersen , John W. Tukey, Scatter/Gather: a cluster-based approach to browsing large document collections, Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval, p.318-329, June 21-24, 1992, Copenhagen, Denmark [doi>10.1145/133160.133214]
• 4: Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., and Harshman, R. A. (1990) Indexing by latent semantic analysis, Journal of the American Society of Information Science, 41(6), pp. 391--407.
• 5: Joachim Diederich , Jörg Kindermann , Edda Leopold , Gerhard Paass, Authorship Attribution with Support Vector Machines, Applied Intelligence, v.19 n.1-2, p.109-123 [doi>10.1023/A:1023824908771]
• 6: Erten, C., Harding, P. J., Kobourov, S. G., Wampler, K., and Yee, G. (2003) Exploring the computing literature using temporal graph visualization, Technical Report, Department of Computer Science, University of Arizona.
• 7: Gray, A., Sallis, P., MacDonell, S. (1997) Software forensics: Extending authorship analysis techniques to computer programs, Proceedings of the 3rd Biannual Conference of the International Association of Forensic Linguists (IAFL), Durham NC.
• 8: Griffiths, T. L., and Steyvers , M. (2004) Finding scientific topics, Proceedings of the National Academy of Sciences, 101 (suppl. 1), 5228--5235.
• 9: Thomas Hofmann, Probabilistic latent semantic indexing, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.50-57, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312649]
• 10: Henry Kautz , Bart Selman , Mehul Shah, Referral Web: combining social networks and collaborative filtering, Communications of the ACM, v.40 n.3, p.63-65, March 1997 [doi>10.1145/245108.245123]
• 11: Krista Lagus , Timo Honkela , Samuel Kaski , Teuvo Kohonen, Websom for Textual Data Mining, Artificial Intelligence Review, v.13 n.5-6, p.345-364, Dec. 1999 [doi>10.1023/A:1006586221250]
• 12: Steve Lawrence , C. Lee Giles , Kurt Bollacker, Digital Libraries and Autonomous Citation Indexing, Computer, v.32 n.6, p.67-71, June 1999 [doi>10.1109/2.769447]
• 13: Andrew McCallum , Kamal Nigam , Lyle H. Ungar, Efficient clustering of high-dimensional data sets with application to reference matching, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.169-178, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347123]
• 14: Mosteller, F., and Wallace, D. (1964) Applied Bayesian and Classical Inference: The Case of the Federalist Papers, Springer-Verlag.
• 15: Mutschke, P. (2003) Mining networks and central entities in digital libraries: a graph theoretic approach applied to co-author networks, Intelligent Data Analysis 2003, Lecture Notes in Computer Science 2810, Springer Verlag, pp. 155--166
• 16: Newman, M. E. J. (2001) Scientific collaboration networks: I. Network construction and fundamental results, Physical Review E, 64, 016131.
• 17: Alexandrin Popescul , Lyle H. Ungar , Gary William Flake , Steve Lawrence , C. Lee Giles, Clustering and Identifying Temporal Trends in Document Databases, Proceedings of the IEEE Advances in Digital Libraries 2000, p.173, May 22-24, 2000
• 18: Michal Rosen-Zvi , Thomas Griffiths , Mark Steyvers , Padhraic Smyth, The author-topic model for authors and documents, Proceedings of the 20th conference on Uncertainty in artificial intelligence, p.487-494, July 07-11, 2004, Banff, Canada
• 19: Thisted, B., and Efron, R. (1987) Did Shakespeare write a newly discovered poem?, Biometrika, pp. 445--455.
• 20: Scott White , Padhraic Smyth, Algorithms for estimating relative importance in networks, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956782]
• 21: Yiming Yang, An Evaluation of Statistical Approaches to Text Categorization, Information Retrieval, v.1 n.1-2, p.69-90, 1999 [doi>10.1023/A:1009982220290]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Algorithms, Experimentation

Keywords:
Gibbs sampling, text modeling, unsupervised learning