Applying co-training methods to statistical parsing

ABSTRACT

We propose a novel Co-Training method for statistical parsing. The algorithm takes as input a small corpus (9695 sentences) annotated with parse trees, a dictionary of possible lexicalized structures for each word in the training set and a large pool of unlabeled text. The algorithm iteratively labels the entire data set with parse trees. Using empirical results based on parsing the Wall Street Journal corpus we show that training a statistical parser on the combined labeled and unlabeled data strongly out-performs training only on the labeled data.

REFERENCES

• 1: E. Black , S. Abney , S. Flickenger , C. Gdaniec , C. Grishman , P. Harrison , D. Hindle , R. Ingria , F. Jelinek , J. Klavans , M. Liberman , M. Marcus , S. Roukos , B. Santorini , T. Strzalkowski, Procedure for quantitatively comparing the syntactic coverage of English grammars, Proceedings of the workshop on Speech and Natural Language, p.306-311, February 19-22, 1991, Pacific Grove, California [doi>10.3115/112405.112467]
• 2: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 3: E. Brill. 1997. Unsupervised learning of disambiguation rules for part of speech tagging. In Natural Language Processing Using Very Large Corpora. Kluwer Academic Press.
• 4: G. Carroll and M. Rooth. 1998. Valence Induction with a Head-Lexicalized PCFG. http://xxx.lanl.gov/abs/cmp-lg/9805001, May.
• 5: Ciprian Chelba , Frederick Jelinek, Exploiting syntactic structure for language modeling, Proceedings of the 36th annual meeting on Association for Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 6: M. Collins and Y. Singer. 1999. Unsupervised Models for Named Entity Classification. In Proc. of WVLC/EMNLP-99, pages 100--110.
• 7: Doug Cutting , Julian Kupiec , Jan Pedersen , Penelope Sibun, A practical part-of-speech tagger, Proceedings of the third conference on Applied natural language processing, March 31-April 03, 1992, Trento, Italy [doi>10.3115/974499.974523]
• 8: David Elworthy, Does Baum-Welch re-estimation help taggers?, Proceedings of the fourth conference on Applied natural language processing, October 13-15, 1994, Stuttgart, Germany [doi>10.3115/974358.974371]
• 9: Eva Wai-man Fong , Dekai Wu, Learning restricted probabilistic link grammars, Connectionist, Statistical, and Symbolic Approaches to Learning for Natural Language Processing, p.173-187, January 1996
• 10: Sally A. Goldman , Yan Zhou, Enhancing Supervised Learning with Unlabeled Data, Proceedings of the Seventeenth International Conference on Machine Learning, p.327-334, June 29-July 02, 2000
• 11: Rebecca Hwa, Sample selection for statistical grammar induction, Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, p.45-52, October 07-08, 2000, Hong Kong [doi>10.3115/1117794.1117800]
• 12: A. K. Joshi and Y. Schabes. 1992. Tree-adjoining grammar and lexicalized grammars. In M. Nivat and A. Podelski, editors, Tree automata and languages, pages 409--431. Elsevier Science.
• 13: A. K. Joshi, L. Levy, and M. Takahashi. 1975. Tree Adjunct Grammars. Journal of Computer and System Sciences.
• 14: A. K. Joshi. 1985. Tree Adjoining Grammars: How much context Sensitivity is required to provide a reasonable structural description. In D. Dowty, I. Karttunen, and A. Zwicky, editors, Natural Language Parsing, pages 206--250. Cambridge University Press, Cambridge, U.K.
• 15: J. Lafferty, D. Sleator, and D. Temperley. 1992. Grammatical trigrams: A probabilistic model of link grammar. In Proc. of the AAAI Conf. on Probabilistic Approaches to Natural Language.
• 16: K. Lari and S. J. Young. 1990. The estimation of stochastic context-free grammars using the Inside-Outside algorithm. Computer Speech and Language, 4:35--56.
• 17: C. de Marcken. 1995. Lexical heads, phrase structure and the induction of grammar. In D. Yarowsky and K. Church, editors, Proc. of 3rd WVLC, pages 14--26, MIT, Cambridge, MA.
• 18: Mitchell P. Marcus , Mary Ann Marcinkiewicz , Beatrice Santorini, Building a large annotated corpus of English: the penn treebank, Computational Linguistics, v.19 n.2, June 1993
• 19: Bernard Merialdo, Tagging English text with a probabilistic model, Computational Linguistics, v.20 n.2, p.155-171, June 1994
• 20: Kamal Nigam , Rayid Ghani, Analyzing the effectiveness and applicability of co-training, Proceedings of the ninth international conference on Information and knowledge management, p.86-93, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354805]
• 21: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 22: Stephen Della Pietra , Vincent J. Della Pietra , J. Gillet , John D. Lafferty , H. Printz , L. Ures, Inference and Estimation of a Long-Range Trigram Model, Proceedings of the Second International Colloquium on Grammatical Inference and Applications, p.78-92, September 21-23, 1994
• 23: A. Ratnaparkhi. 1996. A Maximum Entropy Part-Of-Speech Tagger. In Proc. of EMNLP-96, University of Pennsylvania.
• 24: Philip Resnik, Probabilistic tree-adjoining grammar as a framework for statistical natural language processing, Proceedings of the 14th conference on Computational linguistics, August 23-28, 1992, Nantes, France [doi>10.3115/992133.992135]
• 25: Yves Schabes, Stochastic lexicalized tree-adjoining grammars, Proceedings of the 14th conference on Computational linguistics, August 23-28, 1992, Nantes, France [doi>10.3115/992133.992136]
• 26: B. Srinivas. 1997. Complexity of Lexical Descriptions and its Relevance to Partial Parsing. Ph.D. thesis, Department of Computer and Information Sciences, University of Pennsylvania.
• 27: Fei Xia , Martha Palmer , Aravind Joshi, A uniform method of grammar extraction and its applications, Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, p.53-62, October 07-08, 2000, Hong Kong [doi>10.3115/1117794.1117801]
• 28: David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Proceedings of the 33rd annual meeting on Association for Computational Linguistics, p.189-196, June 26-30, 1995, Cambridge, Massachusetts [doi>10.3115/981658.981684]