Effective multi-label active learning for text classification

ABSTRACT

Labeling text data is quite time-consuming but essential for automatic text classification. Especially, manually creating multiple labels for each document may become impractical when a very large amount of data is needed for training multi-label text classifiers. To minimize the human-labeling efforts, we propose a novel multi-label active learning approach which can reduce the required labeled data without sacrificing the classification accuracy. Traditional active learning algorithms can only handle single-label problems, that is, each data is restricted to have one label. Our approach takes into account the multi-label information, and select the unlabeled data which can lead to the largest reduction of the expected model loss. Specifically, the model loss is approximated by the size of version space, and the reduction rate of the size of version space is optimized with Support Vector Machines (SVM). An effective label prediction method is designed to predict possible labels for each unlabeled data point, and the expected loss for multi-label data is approximated by summing up losses on all labels according to the most confident result of label prediction. Experiments on several real-world data sets (all are publicly available) demonstrate that our approach can obtain promising classification result with much fewer labeled data than state-of-the-art methods.

REFERENCES

• 1: Colin Campbell , Nello Cristianini , Alex J. Smola, Query Learning with Large Margin Classifiers, Proceedings of the Seventeenth International Conference on Machine Learning, p.111-118, June 29-July 02, 2000
• 2: D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. In Advances in Neural Information Processing Systems, volume 7, pages 705--712. The MIT Press, 1995.
• 3: Corinna Cortes , Vladimir Vapnik, Support-Vector Networks, Machine Learning, v.20 n.3, p.273-297, Sept. 1995 [doi>10.1023/A:1022627411411]
• 4: Andrea Esuli , Fabrizio Sebastiani, Active Learning Strategies for Multi-Label Text Classification, Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval, April 06-09, 2009, Toulouse, France [doi>10.1007/978-3-642-00958-7_12]
• 5: R.-E. Fan and C.-J. Lin. A study on threshold selection for multi-label classification. Technical Report, National Taiwan University, 2007.
• 6: T. Joachims. Text categorization with support vector machines: Learning with many relevant features. pages 137--142. Springer Verlag, 1998.
• 7: Thorsten Joachims, Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms, Kluwer Academic Publishers, Norwell, MA, 2002
• 8: H. Kazawa, T. Izumitani, H. Taira, and E. Maeda. Maximal margin labeling for multi-topic text categorization. In Advances in Neural Information Processing Systems (NIPS'05), pages 649--656, 2005.
• 9: K. Brinker. On Active Learning in Multi-label Classification. "FromData and Information Analysis to Knowledge Engineering" of BookSeries "Studiesin Classification, Data Analysis, and Knowledge Organization", Springer, 2006. 1, 2.
• 10: David D. Lewis , William A. Gale, A sequential algorithm for training text classifiers, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, July 03-06, 1994, Dublin, Ireland
• 11: David D. Lewis , Yiming Yang , Tony G. Rose , Fan Li, RCV1: A New Benchmark Collection for Text Categorization Research, The Journal of Machine Learning Research, 5, p.361-397, 12/1/2004
• 12: X. Li, L. Wang, and E. Sung. Multi-label svm active learning for image classification. In International Conference on Image Processing, pages 2207--2210, 2004.
• 13: Hsuan-Tien Lin , Chih-Jen Lin , Ruby C. Weng, A note on Platt's probabilistic outputs for support vector machines, Machine Learning, v.68 n.3, p.267-276, October 2007 [doi>10.1007/s10994-007-5018-6]
• 14: Tong Luo , Kurt Kramer , Dmitry B. Goldgof , Lawrence O. Hall , Scott Samson , Andrew Remsen , Thomas Hopkins, Active Learning to Recognize Multiple Types of Plankton, The Journal of Machine Learning Research, 6, p.589-613, 12/1/2005
• 15: Naonori Ueda , Kazumi Saito, Single-shot detection of multiple categories of text using parametric mixture models, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775140]
• 16: G.-J. Qi, X.-S. Hua, Y. Rui, J. Tang, and H.-J. Zhang. Two-dimensional active learning for image classification. IEEE Conference on Computer Vision and Pattern Recognition, 2008.
• 17: G.-J. Qi, X.-S. Hua, Y. Rui, J. Tang, and H.-J. Zhang. Two-dimensional multi-label active learning with an efficient online adaptation model for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 99(1), 2008.
• 18: Nicholas Roy , Andrew McCallum, Toward Optimal Active Learning through Sampling Estimation of Error Reduction, Proceedings of the Eighteenth International Conference on Machine Learning, p.441-448, June 28-July 01, 2001
• 19: H. S. Seung , M. Opper , H. Sompolinsky, Query by committee, Proceedings of the fifth annual workshop on Computational learning theory, p.287-294, July 27-29, 1992, Pittsburgh, Pennsylvania, United States [doi>10.1145/130385.130417]
• 20: Simon Tong , Daphne Koller, Active learning: theory and applications, 2001
• 21: Simon Tong , Daphne Koller, Support vector machine active learning with applications to text classification, The Journal of Machine Learning Research, 2, p.45-66, 3/1/2002 [doi>10.1162/153244302760185243]
• 22: Rong Yan , Jie Yang , Alexander Hauptmann, Automatically Labeling Video Data Using Multi-class Active Learning, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.516, October 13-16, 2003
• 23: Yiming Yang, A study of thresholding strategies for text categorization, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.137-145, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383975]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

General Terms:
Algorithms, Experimentation, Performance

Keywords:
active learning, multi-label classification, support vector machines, text classification