Scaling to very very large corpora for natural language disambiguation

ABSTRACT

The amount of readily available on-line text has reached hundreds of billions of words and continues to grow. Yet for most core natural language tasks, algorithms continue to be optimized, tested and compared after training on corpora consisting of only one million words or less. In this paper, we evaluate the performance of different learning methods on a prototypical natural language disambiguation task, confusion set disambiguation, when trained on orders of magnitude more labeled data than has previously been used. We are fortunate that for this particular application, correctly labeled training data is free. Since this will often not be the case, we examine methods for effectively exploiting very large corpora when labeled data comes at a cost.

REFERENCES

• 1: Michele Banko , Eric Brill, Mitigating the paucity-of-data problem: exploring the effect of training corpus size on classifier performance for natural language processing, Proceedings of the first international conference on Human language technology research, p.1-5, March 18-21, 2001, San Diego [doi>10.3115/1072133.1072204]
• 2: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350]
• 3: Eric Brill , Jun Wu, Classifier combination for improved lexical disambiguation, Proceedings of the 17th international conference on Computational linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 4: Charniak, E. (1996). Treebank Grammars, Proceedings AAAI-96, Menlo Park, Ca.
• 5: Dagan, I. and Engelson, S. (1995). Committee-based sampling for training probabilistic classifiers. In Proc. ML-95, the 12th Int. Conf. on Machine Learning.
• 6: Gale, W. A., Church, K. W., and Yarowsky, D. (1993). A method for disambiguating word senses in a large corpus. Computers and the Humanities, 26:415--439.
• 7: Golding, A. R. (1995). A Bayesian hybrid method for context-sensitive spelling correction. In Proc. 3rd Workshop on Very Large Corpora, Boston, MA.
• 8: Andrew R. Golding , Dan Roth, A Winnow-Based Approach to Context-Sensitive Spelling Correction, Machine Learning, v.34 n.1-3, p.107-130, Feb. 1999 [doi>10.1023/A:1007545901558]
• 9: Andrew R. Golding , Yves Schabes, Combining Trigram-based and feature-based methods for context-sensitive spelling correction, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.71-78, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981873]
• 10: Henderson, J. C. and Brill, E (1999). Exploiting diversity in natural language processing: combining parsers. In 1999 Joint Sigdat Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. ACL, New Brunswick NJ. 187--194.
• 11: Michael P. Jones , James H. Martin, Contextual spelling correction using latent semantic analysis, Proceedings of the fifth conference on Applied natural language processing, p.166-173, March 31-April 03, 1997, Washington, DC [doi>10.3115/974557.974582]
• 12: Lewis, D. D., & Catlett, J. (1994). Heterogeneous uncertainty sampling. Proceedings of the Eleventh International Conference on Machine Learning (pp. 148--156). New Brunswick, NJ: Morgan Kaufmann.
• 13: Lidia Mangu , Eric Brill, Automatic Rule Acquisition for Spelling Correction, Proceedings of the Fourteenth International Conference on Machine Learning, p.187-194, July 08-12, 1997
• 14: Bernard Merialdo, Tagging English text with a probabilistic model, Computational Linguistics, v.20 n.2, p.155-171, June 1994
• 15: Mitchell, T. M. (1999), The role of unlabeled data in supervised learning, in Proceedings of the Sixth International Colloquium on Cognitive Science, San Sebastian, Spain.
• 16: Kamal Nigam , Andrew McCallum , Sebastian Thrun , Tom Mitchell, Learning to classify text from labeled and unlabeled documents, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.792-799, July 1998, Madison, Wisconsin, United States
• 17: Ted Pedersen, A simple approach to building ensembles of Naive Bayesian classifiers for word sense disambiguation, Proceedings of the first conference on North American chapter of the Association for Computational Linguistics, p.63-69, April 29-May 04, 2000, Seattle, Washington
• 18: Powers, D. (1997). Learning and application of differential grammars. In Proc. Meeting of the ACL Special Interest Group in Natural Language Learning, Madrid.
• 19: Hans van Halteren , Jakub Zavrel , Walter Daelemans, Improving data driven wordclass tagging by system combination, Proceedings of the 36th annual meeting on Association for Computational Linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 20: Weng, F., Stolcke, A, & Sankar, A (1998). Efficient lattice representation and generation. Proc. Intl. Conf. on Spoken Language Processing, vol. 6, pp. 2531--2534. Sydney, Australia.
• 21: David Yarowsky, Decision lists for lexical ambiguity resolution: application to accent restoration in Spanish and French, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.88-95, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981745]
• 22: David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Proceedings of the 33rd annual meeting on Association for Computational Linguistics, p.189-196, June 26-30, 1995, Cambridge, Massachusetts [doi>10.3115/981658.981684]