Cross-training: learning probabilistic mappings between topics

ABSTRACT

Classification is a well-established operation in text mining. Given a set of labels A and a set D_A of training documents tagged with these labels, a classifier learns to assign labels to unlabeled test documents. Suppose we also had available a different set of labels B, together with a set of documents D_B marked with labels from B. If A and B have some semantic overlap, can the availability of D_B help us build a better classifier for A, and vice versa? We answer this question in the affirmative by proposing cross-training: a new approach to semi-supervised learning in presence of multiple label sets. We give distributional and discriminative algorithms for cross-training and show, through extensive experiments, that cross-training can discover and exploit probabilistic relations between two taxonomies for more accurate classification.

REFERENCES

• 1: Rakesh Agrawal , Ramakrishnan Srikant, On integrating catalogs, Proceedings of the 10th international conference on World Wide Web, p.603-612, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372163]
• 2: J. Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12:149--198, 2000. http://www-2.cs.cmu.edu/afs/cs/project/jair/pub/volume12/baxterOOa.pdf.
• 3: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 4: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 5: A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, B(39):1--38, 1977.
• 6: AnHai Doan , Jayant Madhavan , Pedro Domingos , Alon Halevy, Learning to map between ontologies on the semantic web, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511532]
• 7: Susan Dumais , John Platt , David Heckerman , Mehran Sahami, Inductive learning algorithms and representations for text categorization, Proceedings of the seventh international conference on Information and knowledge management, p.148-155, November 02-07, 1998, Bethesda, Maryland, United States [doi>10.1145/288627.288651]
• 8: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 9: Wen-Syan Li , Quoc Vu , Divakant Agrawal , Yoshinori Hara , Hajime Takano, PowerBookmarks: a system for personalizable Web information organization, sharing, and management, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1375-1389, May 17, 1999
• 10: Bing Liu , Wee Sun Lee , Philip S. Yu , Xiaoli Li, Partially Supervised Classification of Text Documents, Proceedings of the Nineteenth International Conference on Machine Learning, p.387-394, July 08-12, 2002
• 11: Yoelle S. Maarek , Israel Z. Ben Shaul, Automatically organizing bookmarks per contents, Proceedings of the fifth international World Wide Web conference on Computer networks and ISDN systems, p.1321-1333, May 1996, Paris, France
• 12: A. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. Software available from http://www.cs.cmu.edu/~mccallum/bow/,1998.
• 13: A. McCallum and K. Nigam. A comparison of event models for naive Bayes text classification. In AAAI/ICML-98 Workshop on Learning for Text Categorization, pages 41--48. AAAI Press, 1998. Online at http://www.cs.cmu.edu/~knigam/.
• 14: K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61--67, 1999. See http://www.cs.cmu.edu/~knigam/ and http://www.cs.cmu.edu/~mccallum/papers/maxent-ijcaiws99.ps.gz.
• 15: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 16: J. Platt. Sequential minimal optimization: A fast algorithm for training support vector machines. Technical Report MSR-TR-98-14, Microsoft Research, 1998. Online at http://www.research.microsoft.com/users/jplatt/smoTR.pdf.
• 17: E. S. Ristad. A natural law of succession. Research report CS-TR-495-95, Princeton University, July 1995.
• 18: S. Thrun and J. O'Sullivan. Discovering structure in multiple learning tasks: The TC algorithm. In L. Saitta, editor, Proceedings of the 13th International Conference on Machine Learning ICML-96, San Mateo, CA, 1996. Morgen Kaufmann.
• 19: Yiming Yang , Xin Liu, A re-examination of text categorization methods, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.42-49, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312647]
• 20: Hwanjo Yu , Jiawei Han , Kevin Chen-Chuan Chang, PEBL: positive example based learning for Web page classification using SVM, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775083]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

Keywords:
EM, document classification, semi-supervised multi-task learning, support vector machines