Learning from labeled features using generalized expectation criteria

ABSTRACT

It is difficult to apply machine learning to new domains because often we lack labeled problem instances. In this paper, we provide a solution to this problem that leverages domain knowledge in the form of affinities between input features and classes. For example, in a baseball vs. hockey text classification problem, even without any labeled data, we know that the presence of the word puck is a strong indicator of hockey. We refer to this type of domain knowledge as a labeled feature. In this paper, we propose a method for training discriminative probabilistic models with labeled features and unlabeled instances. Unlike previous approaches that use labeled features to create labeled pseudo-instances, we use labeled features directly to constrain the model's predictions on unlabeled instances. We express these soft constraints using generalized expectation (GE) criteria --- terms in a parameter estimation objective function that express preferences on values of a model expectation. In this paper we train multinomial logistic regression models using GE criteria, but the method we develop is applicable to other discriminative probabilistic models. The complete objective function also includes a Gaussian prior on parameters, which encourages generalization by spreading parameter weight to unlabeled features. Experimental results on text classification data sets show that this method outperforms heuristic approaches to training classifiers with labeled features. Experiments with human annotators show that it is more beneficial to spend limited annotation time labeling features rather than labeling instances. For example, after only one minute of labeling features, we can achieve 80% accuracy on the ibm vs. mac text classification problem using GE-FL, whereas ten minutes labeling documents results in an accuracy of only 77%

REFERENCES

• 1: David M. Blei , Andrew Y. Ng , Michael I. Jordan, Latent dirichlet allocation, The Journal of Machine Learning Research, 3, p.993-1022, 3/1/2003 [doi>10.1162/jmlr.2003.3.4-5.993]
• 2: M. Chang, L. Ratinov, and D. Roth. Guiding semi-supervision with constraint-driven learning. In ACL, 2007.
• 3: David Cohn , Les Atlas , Richard Ladner, Improving Generalization with Active Learning, Machine Learning, v.15 n.2, p.201-221, May 1994 [doi>10.1023/A:1022673506211]
• 4: Aynur Dayanik , David D. Lewis , David Madigan , Vladimir Menkov , Alexander Genkin, Constructing informative prior distributions from domain knowledge in text classification, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148255]
• 5: Yoav Freund , H. Sebastian Seung , Eli Shamir , Naftali Tishby, Selective Sampling Using the Query by Committee Algorithm, Machine Learning, v.28 n.2-3, p.133-168, Aug./Sept. 1997 [doi>10.1023/A:1007330508534]
• 6: Shantanu Godbole , Abhay Harpale , Sunita Sarawagi , Soumen Chakrabarti, Document classification through interactive supervision of document and term labels, Proceedings of the 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, p.185-196, September 20-24, 2004, Pisa, Italy
• 7: J. Graca, K. Ganchev, and B. Taskar. Expectation maximization and posterior constraints. In NIPS, 2007.
• 8: Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2004.
• 9: Aria Haghighi , Dan Klein, Prototype-driven learning for sequence models, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.320-327, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220876]
• 10: Yifen Huang , Tom M. Mitchell, Text clustering with extended user feedback, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148242]
• 11: R. Jin and Y. Liu. A framework for incorporating class priors into discriminative classification. In PAKDD, 2005.
• 12: Thorsten Joachims, Transductive Inference for Text Classification using Support Vector Machines, Proceedings of the Sixteenth International Conference on Machine Learning, p.200-209, June 27-30, 1999
• 13: D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In ICML, 1994.
• 14: B. Liu, X. Li, W. Lee, and P. Yu. Text classification by labeling words. In AAAI, 2004.
• 15: Gideon S. Mann , Andrew McCallum, Simple, robust, scalable semi-supervised learning via expectation regularization, Proceedings of the 24th international conference on Machine learning, p.593-600, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273571]
• 16: A. McCallum, G. Mann, and G. Druck. Generalized expectation criteria. Technical Report 2007-62, University of Massachusetts, Amherst, 2007.
• 17: Hema Raghavan , James Allan, An interactive algorithm for asking and incorporating feature feedback into support vector machines, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277758]
• 18: Hema Raghavan , Omid Madani , Rosie Jones, Active Learning with Feedback on Features and Instances, The Journal of Machine Learning Research, 7, p.1655-1686, 12/1/2006
• 19: Robert E. Schapire , Marie Rochery , Mazin G. Rahim , Narendra Gupta, Incorporating Prior Knowledge into Boosting, Proceedings of the Nineteenth International Conference on Machine Learning, p.538-545, July 08-12, 2002
• 20: Xiaoyun Wu , Rohini Srihari, Incorporating prior knowledge with weighted margin support vector machines, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014089]
• 21: X. Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
labeled features, learning with domain knowledge, semi-supervised learning, text classification