Document centered approach to text normalization

ABSTRACT

In this paper we present an approach to tackle three important problems of text normalization: sentence boundary disambiguation, disambiguation of capitalized words when they are used in positions where capitalization is expected, and identification of abbreviations. The main feature of our approach is that it uses a minimum of pre-built resources, instead dynamically inferring disambiguation clues from the entire document itself. This makes it domain independent, closely targeted to each individual document and portable to other languages. We thoroughly evaluated this approach on several corpora and it showed high accuracy.

REFERENCES

• 1: John Aberdeen , John Burger , David Day , Lynette Hirschman , Patricia Robinson , Marc Vilain, MITRE: description of the Alembic system used for MUC-6, Proceedings of the 6th conference on Message understanding, November 06-08, 1995, Columbia, Maryland [doi>10.3115/1072399.1072413]
• 2: B. Baldwin, C. Doran, J. Reynar, M. Niv, B. Srinivas and M. Wasson. Eagle: An extensible architecture for general linguistic engineering. In Proceedings of RIAO '97, Montreal, June 1997.
• 3: Kenneth Ward Church, One term or two?, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.310-318, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215376]
• 4: P. Clarkson , A. Robinson, Language Model Adaptation Using Mixtures and an Exponentially Decaying Cache, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2, p.799, April 21-24, 1997
• 5: William A. Gale , Kenneth W. Church , David Yarowsky, One sense per discourse, Proceedings of the workshop on Speech and Natural Language, February 23-26, 1992, Harriman, New York [doi>10.3115/1075527.1075579]
• 6: R. Kuhn , R. De Mori, A Cache-Based Natural Language Model for Speech Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.6, p.570-583, June 1990 [doi>10.1109/34.56193]
• 7: Inderjeet Mani , T. Richard MacMillan, Identifying unknown proper names in newswire text, Corpus processing for lexical acquisition, MIT Press, Cambridge, MA, 1996
• 8: Mitchell P. Marcus , Mary Ann Marcinkiewicz , Beatrice Santorini, Building a large annotated corpus of English: the penn treebank, Computational Linguistics, v.19 n.2, June 1993
• 9: Andrei Mikheev, Automatic rule induction for unknown-word guessing, Computational Linguistics, v.23 n.3, p.405-423, September 1997
• 10: Andrei Mikheev, A knowledge-free method for capitalized word disambiguation, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.159-166, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034710]
• 11: David D. Palmer , Marti A. Hearst, Adaptive multilingual sentence boundary disambiguation, Computational Linguistics, v.23 n.2, p.241-267, June 1997
• 12: Michael D. Riley, Some applications of tree-based modelling to speech and language, Proceedings of the workshop on Speech and Natural Language, October 15-18, 1989, Cape Cod, Massachusetts [doi>10.3115/1075434.1075492]
• 13: K. Seymore, S. Chen and R. Rosenfeld. Nonlinear interpolation of topic models for language model adaptation. In Proceedings of ICSLP98, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing

Additional Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation
          Subjects: Languages and systems

General Terms:
Design, Experimentation