Topic model methods for automatically identifying out-of-scope resources

ABSTRACT

Recent years have seen the rise of subject-themed digital libraries, such as the NSDL pathways and the Digital Library for Earth System Education (DLESE). These libraries often need to manually verify that contributed resources cover topics that fit within the theme of the library. We show that such scope judgments can be automated using a combination of text classification techniques and topic modeling. Our models address two significant challenges in making scope judgments: only a small number of out-of-scope resources are typically available, and the topic distinctions required for digital libraries are much more subtle than classic text classification problems. To meet these challenges, our models combine support vector machine learners optimized to different performance metrics and semantic topics induced by unsupervised statistical topic models. Our best model is able to distinguish resources that belong in DLESE from resources that don't with an accuracy of around 70%. We see these models as the first steps towards increasing the scalability of digital libraries and dramatically reducing the workload required to maintain them.

REFERENCES

• 1: Alias-i. LingPipe 3.7.0. http://alias-i.com/lingpipe/, Oct. 2008.
• 2: Robert B. Allen , Yejun Wu, Metrics for the scope of a collection: Research Articles, Journal of the American Society for Information Science and Technology, v.56 n.12, p.1243-1249, October 2005 [doi>10.1002/asi.v56:12]
• 3: BEN. BiosciEdNet. http://www.biosciednet.org/, 2009.
• 4: D. Blei and J. McAuliffe. Supervised topic models. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 121--128. MIT Press, Cambridge, MA, 2008.
• 5: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 6: I. Buciu. Learning sparse non-negative features for object recognition. In Intelligent Computer Communication and Processing, 2007 IEEE International Conference on, pages 73--79, 2007.
• 7: comPADRE. Resources for physics and astronomy education. http://www.compadre.org/, 2009.
• 8: DLESE. Digital library for earth system education. http://www.dlese.org/, 2009.
• 9: H. Drucker, D. Wu, and V. Vapnik. Support vector machines for spam categorization. Neural Networks, IEEE Transactions on, 10(5):1048--1054, 1999.
• 10: P. Ginsparg. Winners and losers in the global research village. In Proceedings of the Joint ICSU Press/UNESCO Expert Conference on Electronic Publishing in Science, 1996.
• 11: D. Guillamet , J. Vitrià, Evaluation of distance metrics for recognition based on non-negative matrix factorization, Pattern Recognition Letters, v.24 n.9-10, p.1599-1605, 01 June 2003 [doi>10.1016/S0167-8655(02)00399-9]
• 12: D. Hiom. The social science information gateway: putting theory into practice. Information Research, 4(1), 1998.
• 13: Lesly Huxley, Follow the Fox to Renardus: An Academic Subject Gateway Service for Europe, Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, p.395-398, September 18-20, 2000
• 14: Thorsten Joachims, Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms, Kluwer Academic Publishers, Norwell, MA, 2002
• 15: Thorsten Joachims, A support vector method for multivariate performance measures, Proceedings of the 22nd international conference on Machine learning, p.377-384, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102399]
• 16: Aleksander Kolcz , Wen-tau Yih, Raising the baseline for high-precision text classifiers, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281237]
• 17: D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788--791, Oct. 1999.
• 18: D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In NIPS, pages 556--562, 2000.
• 19: David D. Lewis , Yiming Yang , Tony G. Rose , Fan Li, RCV1: A New Benchmark Collection for Text Categorization Research, The Journal of Machine Learning Research, 5, p.361-397, 12/1/2004
• 20: Yaoyong Li , Kalina Bontcheva , Hamish Cunningham, Adapting svm for data sparseness and imbalance: A case study in information extraction, Natural Language Engineering, v.15 n.2, p.241-271, April 2009 [doi>10.1017/S1351324908004968]
• 21: Y. Li and J. Shawe-Taylor. The SVM with uneven margins and chinese document categorization. COLIPS PUBLICATIONS, 2003.
• 22: David Mimno , Andrew McCallum, Organizing the OCA: learning faceted subjects from a library of digital books, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255249]
• 23: A. Moschitti and R. Basili. Complex Linguistic Features for Text Classification: A Comprehensive Study, pages 181--196. Springer Berlin / Heidelberg, 2004.
• 24: David Newman , Kat Hagedorn , Chaitanya Chemudugunta , Padhraic Smyth, Subject metadata enrichment using statistical topic models, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255248]
• 25: Kanal Paul Nigam , Tom M. Mitchell, Using unlabeled data to improve text classification, Carnegie Mellon University, Pittsburgh, PA, 2001
• 26: NSDL. National science digital library. http://nsdl.org/, 2009.
• 27: OAIster. Open archives initiative (OAI)ster. http://www.oaister.org/, 2009.
• 28: OCA. Open content alliance. http://www.opencontentalliance.org/, 2009.
• 29: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 30: P. Soucy. Beyond TFIDF weighting for text categorization in the vector space model. In Proceedings of the Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005, pages 1130--1135, 2005.
• 31: D. Soukup and I. Bajla. Robust object recognition under partial occlusions using NMF. Computational Intelligence and Neuroscience, 2008:857453, 2008\vadjust\newpage. PMC2396239.
• 32: M. Steyvers and T. Griffiths. Probabilistic topic models. In T. Landauer, Mc, S. Dennis, and W. Kintsch, editors, Latent Semantic Analysis: A Road to Meaning. Lawrence Earlbaum, 2007.
• 33: Sriharsha Veeramachaneni , Diego Sona , Paolo Avesani, Hierarchical Dirichlet model for document classification, Proceedings of the 22nd international conference on Machine learning, p.928-935, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102468]
• 34: Ozgur Yilmazel , Niranjan Balasubramanian , Sarah C. Harwell , Jennifer Bailey , Anne R. Diekema , Elizabeth D. Liddy, Text Categorization for Aligning Educational Standards, Proceedings of the 40th Annual Hawaii International Conference on System Sciences, p.73, January 03-06, 2007 [doi>10.1109/HICSS.2007.517]
• 35: S. Zhou, K. Li, and Y. Liu. Text Categorization Based on Topic Model, pages 572--579. Springer Berlin / Heidelberg, 2008.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Algorithms

Keywords:
digital libraries, machine learning, relevance, scope, topics