Topic models and a revisit of text-related applications

ABSTRACT

Topic models such as aspect model or LDA have been shown as a promising approach for text modeling. Unlike many previous models that restrict each document to a single topic, topic models support the important idea that each document could be relevant to multiple topics. This makes topic models significantly more expressive in modeling text documents. However, we observe two limitations in topic models. One is that of scalability as it is extremely expensive to run the models on large corpora. The other limitation is the inability to model the key concept of relevance. This prevents the models from being directly applied to goals such as text classification and relevance feedback for query modification; in these goals, items relevant to topics (classes and queries) are provided upfront. The first aim of this paper is to sketch solutions for these limitations. To alleviate the scalability problem, we introduce a one-scan topic model requiring only a single pass over a corpus for inference. To overcome the latter, we propose relevance-based topic models that have the advantages of previous models while taking the concept of relevance into account. The second aim, based on the proposed models, is to revisit a wide range of well-known but still open text-related tasks, and outline our vision on how the approaches for the tasks could be improved by topic models.

REFERENCES

• 1: Adrieu, C., Freitas, N., Doucet, A., Jordan, M., An Introduction to Markov Chain Monte Carlo for Machine Learning, Machine Learning, 50, 2003.
• 2: Bhamidipati, N., Pal, S., Stemming via Distribution-based Word Segregation for Classification and Retrieval, In IEEE Transactions on Systems, Man, and Cybernetics, 37(2), 2007.
• 3: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003
• 4: Bradley P. S., Fayyad, U., Reina, C., Scaling Clustering Algorithms to Large Databases, In Proceedings of the 4th ACM SIG International Conference on Knowledge Discovery and Data Mining Conference (KDD), 1998.
• 5: Erosheva, E., Fienberg, S., Lafferty, J., Mixed-membership Models of Scientific Publication, In Proceedings of National Academy of Science (PNAS), 2004.
• 6: Farnstrom, F. Lewis, J., Elkan, C., Scalability for Clustering Algorithms Revisited, In Proceedings of the 6th ACM SIG International Conference on Knowledge Discovery and Data Mining Conference (KDD), 2000.
• 7: Griffiths, T., Steyvers, M., Finding Scientific Topics, In Proceedings of National Academy of Science (PNAS), 2004.
• 8: Ha-Thuc, V., Nguyen, D. C., Srinivasan, P., A Quality-Threshold Data Summarization Algorithm, In Proceedings of the 6th IEEE International Conference on Research, Innovation and Vision for the Future (RIVF), 2008.
• 9: Djoerd Hiemstra , Stephen Robertson , Hugo Zaragoza, Parsimonious language models for information retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009025]
• 10: Hofmann, T., Probabilistic Latent Semantic Indexing, In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence (UAI), 1999.
• 11: Lauser, B., Hotho, A., Automatic Multi-label Subject Indexing in a Multi-lingual Environment, In Proceedings of the 7th European Conference in Research and Advanced Technology for Digital Libraries (ECDL), 2003.
• 12: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 13: Xiaoyong Liu , W. Bruce Croft, Cluster-based retrieval using language models, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009026]
• 14: McCallum, A., Multi-Label Text Classification with a Mixture Model Trained by EM, In Proceedings of AAAI Workshop on Text Learning, 1999.
• 15: Andrew Kachites McCallum , Kamal Nigam , Jason Rennie , Kristie Seymore, Automating the Construction of Internet Portals with Machine Learning, Information Retrieval, v.3 n.2, p.127-163, July 2000 [doi>10.1023/A:1009953814988]
• 16: Kamal Nigam , Rayid Ghani, Analyzing the effectiveness and applicability of co-training, Proceedings of the ninth international conference on Information and knowledge management, p.86-93, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354805]
• 17: Robertson, S., Sparck-Jones, K., Relevance Weighting of Search Terms, Journal of American Society for Information Science, 27, 1988.
• 18: Sparck-Jones, A., Robertson, S., Hiemstra, D., Zaragoza, H., Language Modelling and Relevance, In Croft, B., and Lafferty, J. (eds.) Language Modeling for Information Retrieval, Kluwer Academics, 2003.
• 19: Steyvers, M., Griffiths, T., Probabilistic Topic Models, In Landauer et al. (eds.) Latent Semantic Analysis: A Road to Meaning, Laurence Erlbaum, 2006.
• 20: Bin Tan , Atulya Velivelli , Hui Fang , ChengXiang Zhai, Term feedback for information retrieval with language models, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277788]
• 21: Xing Wei , W. Bruce Croft, LDA-based document models for ad-hoc retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148204]
• 22: Jinxi Xu , W. Bruce Croft, Corpus-based stemming using cooccurrence of word variants, ACM Transactions on Information Systems (TOIS), v.16 n.1, p.61-81, Jan. 1998 [doi>10.1145/267954.267957]
• 23: Ding Zhou , Xiang Ji , Hongyuan Zha , C. Lee Giles, Topic evolution and social interactions: how authors effect research, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183653]
• 24: Ding Zhou , Eren Manavoglu , Jia Li , C. Lee Giles , Hongyuan Zha, Probabilistic models for discovering e-communities, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135807]
• 25: http://www.dcs.gla.ac.uk/idom/ir_resources/linguistic_utils/stop_words

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.0 General

General Terms:
Algorithms, Design, Experimentation, Languages, Performance, Theory

Keywords:
gibbs sampling, lda, relevance-based language models, topic models