Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

We present a new semi-supervised training procedure for conditional random fields (CRFs) that can be used to train sequence segmentors and labelers from a combination of labeled and unlabeled training data. Our approach is based on extending the minimum entropy regularization framework to the structured prediction case, yielding a training objective that combines unlabeled conditional entropy with labeled conditional likelihood. Although the training objective is no longer concave, it can still be used to improve an initial model (e.g. obtained from supervised training) by iterative ascent. We apply our new training algorithm to the problem of identifying gene and protein mentions in biological texts, and show that incorporating unlabeled data improves the performance of the supervised CRF in this case.

REFERENCES

• 1: Steven Abney, Understanding the Yarowsky Algorithm, Computational Linguistics, v.30 n.3, p.365-395, September 2004 [doi>10.1162/0891201041850876]
• 2: Y. Altun, D. McAllester and M. Belkin. (2005). Maximum margin semi-supervised learning for structured variables. Advances in Neural Information Processing Systems 18.
• 3: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 4: Stephen Boyd , Lieven Vandenberghe, Convex Optimization, Cambridge University Press, New York, NY, 2004
• 5: V. Castelli and T. Cover. (1996). The relative value of labeled and unlabeled samples in pattern recognition with an unknown mixing parameter. IEEE Trans. on Information Theory, 42(6):2102--2117.
• 6: Gilles Celeux , Gérard Govaert, A Classification EM algorithm for clustering and two stochastic versions, Computational Statistics & Data Analysis, v.14 n.3, p.315-332, Oct. 1992 [doi>10.1016/0167-9473(92)90042-E]
• 7: I. Cohen and F. Cozman. (2006). Risks of semi-supervised learning. Semi-Supervised Learning, O. Chapelle, B. Scholköpf and A. Zien, (Editors), 55--70, MIT Press.
• 8: A. Corduneanu and T. Jaakkola. (2006). Data dependent regularization. Semi-Supervised Learning, O. Chapelle, B. Scholköpf and A. Zien, (Editors), 163--182, MIT Press.
• 9: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 10: R. Duda and P. Hart. (1973). Pattern Classification and Scene Analysis, John Wiley & Sons.
• 11: Y. Grandvalet and Y. Bengio. (2004). Semi-supervised learning by entropy minimization, Advances in Neural Information Processing Systems, 17:529--536.
• 12: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 13: W. Li and A. McCallum. (2005). Semi-supervised sequence modeling with syntactic topic models. Proceedings of Twentieth National Conference on Artificial Intelligence, 813--818.
• 14: A. McCallum. (2002). MALLET: A machine learning for language toolkit. {http://mallet.cs.umass.edu}
• 15: R. McDonald, K. Lerman and Y. Jin. (2005). Conditional random field biomedical entity tagger. {http://www.seas.upenn.edu/~sryantm/software/BioTagger/}
• 16: R. McDonald and F. Pereira. (2005). Identifying gene and protein mentions in text using conditional random fields. BMC Bioinformatics 2005, 6(Suppl 1):S6.
• 17: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 18: J. Nocedal and S. Wright. (2000). Numerical Optimization, Springer.
• 19: S. Roberts, R. Everson and I. Rezek. (2000). Maximum certainty data partitioning. Pattern Recognition, 33(5):833--839.
• 20: S. V. N. Vishwanathan , Nicol N. Schraudolph , Mark W. Schmidt , Kevin P. Murphy, Accelerated training of conditional random fields with stochastic gradient methods, Proceedings of the 23rd international conference on Machine learning, p.969-976, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143966]
• 21: David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Proceedings of the 33rd annual meeting on Association for Computational Linguistics, p.189-196, June 26-30, 1995, Cambridge, Massachusetts [doi>10.3115/981658.981684]
• 22: D. Zhou, O. Bousquet, T. Navin Lal, J. Weston and B. Schölkopf. (2004). Learning with local and global consistency. Advances in Neural Information Processing Systems, 16:321--328.
• 23: Dengyong Zhou , Jiayuan Huang , Bernhard Schölkopf, Learning from labeled and unlabeled data on a directed graph, Proceedings of the 22nd international conference on Machine learning, p.1036-1043, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102482]
• 24: X. Zhu, Z. Ghahramani and J. Lafferty. (2003). Semisupervised learning using Gaussian fields and harmonic functions. Proceedings of the 20th International Conference on Machine Learning, 912--919.