Unsupervised estimation for noisy-channel models

ABSTRACT

Shannon's Noisy-Channel model, which describes how a corrupted message might be reconstructed, has been the corner stone for much work in statistical language and speech processing. The model factors into two components: a language model to characterize the original message and a channel model to describe the channel's corruptive process. The standard approach for estimating the parameters of the channel model is unsupervised Maximum-Likelihood of the observation data, usually approximated using the Expectation-Maximization (EM) algorithm. In this paper we show that it is better to maximize the joint likelihood of the data at both ends of the noisy-channel. We derive a corresponding bi-directional EM algorithm and show that it gives better performance than standard EM on two tasks: (1) translation using a probabilistic lexicon and (2) adaptation of a part-of-speech tagger between related languages.

REFERENCES

• 1: Lalit R. Bahl , Frederick Jelinek , Robert L. Mercer, A maximum likelihood approach to continuous speech recognition, Readings in speech recognition, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1990
• 2: Baum, L., Peterie, T., Souled, G., & Weiss, N. (1970). A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains. Ann. Math. Statist., 41, 164--171.
• 3: Thorsten Brants, TnT: a statistical part-of-speech tagger, Proceedings of the sixth conference on Applied natural language processing, p.224-231, April 29-May 04, 2000, Seattle, Washington [doi>10.3115/974147.974178]
• 4: P. Brown , J. Cocke , S. Della Pietra , V. Della Pietra , F. Jelinek , R. Mercer , P. Roossin, A statistical approach to language translation, Proceedings of the 12th conference on Computational linguistics, p.71-76, August 22-27, 1988, Budapest, Hungry [doi>10.3115/991635.991651]
• 5: Clarkson, P., & Rosenfeld, R. (1997). Statistical language modeling using the cmu-cambridge toolkit. Proceedings ESCA Eurospeech.
• 6: Daelemans, W., Zavrel, J., Berck, P., & Gillis, S. (1996). Mbt: A memory-based part of speech tagger generator. Proceedings of the fourth Workshop on Very Large Corpora (ACL SIGDAT) (pp. 14--27). Copenhagen, Denmark.
• 7: Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, Series B, 39, 1--38.
• 8: Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. MT Summit.
• 9: Philipp Koehn , Kevin Knight, Estimating Word Translation Probabilities from Unrelated Monolingual Corpora Using the EM Algorithm, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.711-715, July 30-August 03, 2000
• 10: Philipp Koehn , Franz Josef Och , Daniel Marcu, Statistical phrase-based translation, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.48-54, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073462]
• 11: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 12: Percy Liang , Ben Taskar , Dan Klein, Alignment by agreement, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.104-111, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220849]
• 13: Maamouri, M., Bies, A., Buckwalter, T., & Mekki, W. (2004). The penn arabic treebank: Building a large-scale annotated arabic corpus. Proceedings of NEMLAR 2004..
• 14: Franz Josef Och , Hermann Ney, A systematic comparison of various statistical alignment models, Computational Linguistics, v.29 n.1, p.19-51, March 2003 [doi>10.1162/089120103321337421]
• 15: Kishore Papineni , Salim Roukos , Todd Ward , Wei-Jing Zhu, BLEU: a method for automatic evaluation of machine translation, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073135]
• 16: Rambow, O., Chiang, D., Diab, M., Habash, N., Hwa, R., Sima'an, K., Lacey, V., Levy, R., Nichols, C., & Shareef, S. (2005). Parsing arabic dialects (Technical Report). Johns Hopkins University 2005 Summer Workshop on Language Engineering.
• 17: Ratnaparkhi, A. (1996). A maximum entropy model for part-of-speech tagging. Proceedings of the Conference on Empirical Methods in Natural Language Processing.
• 18: Claude E. Shannon , Warren Weaver, A Mathematical Theory of Communication, University of Illinois Press, Champaign, IL, 1963
• 19: Richard Zens , Evgeny Matusov , Hermann Ney, Improved word alignment using a symmetric lexicon model, Proceedings of the 20th international conference on Computational Linguistics, p.36-es, August 23-27, 2004, Geneva, Switzerland [doi>10.3115/1220355.1220361]