Successfully detecting and correcting false friends using channel profiles

ABSTRACT

The detection and correction of false friends - also called real-word errors - is a notoriously difficult problem. On realistic data, the break-even point for automatic correction so far could not be reached: the number of additional infelicitous corrections outnumbered the useful corrections. We present a new approach where we first compute a profile of the error channel for the given text. During the correction process, the profile helps to restrict attention to a small set of "suspicious" lexical tokens of the input text where it is "plausible" to assume that the token represents a false friend. Using a conventional word trigram statistics for disambiguation we obtain a correction method that can be successfully applied to unrestricted text. In experiments for OCR documents, we show significant accuracy gains by fully automatic correction of false friends.

REFERENCES

• 1: I. A. Bolshakov and A. F. Gelbukh. On detection of malapropisms by multistage collocation testing. In 8th International Conference on Applications of Natural Language to Information Systems, June 2003, Burg (Spreewald), Germany, pages 28--41, 2003.
• 2: T. Brants and A. Franz. Web 1t 5-gram version 1. Linguistic Data Consortium, Philadelphia, 2006.
• 3: A. Dengel, R. Hoch, F. Hönes, T. Jäger, M. Malburg, and A. Weigel. Techniques for improving OCR results. In H. Bunke and P. S. Wang, editors, Handbook of Character Recognition and Document Image Analysis, pages 227--258. World Scientific, 1997.
• 4: W. A. Gale, K. W. Church, and D. Yarowsky. Discrimination decisions for 100,000-dimensional spaces. Current Issues in Computational Linguistics: In Honour of Don Walker, pages 429--450, 1994.
• 5: C. Giuliano. jWeb1T: a library for searching the Web 1T 5-gram corpus, 2007. Software available at http://tcc.itc.it/research/textec/tools-resources/jweb1t.html.
• 6: A. R. Golding. A bayesian hybrid method for context-sensitive spelling correction. pages 39--53, 1995.
• 7: Andrew R. Golding , Dan Roth, A Winnow-Based Approach to Context-Sensitive Spelling Correction, Machine Learning, v.34 n.1-3, p.107-130, Feb. 1999 [doi>10.1023/A:1007545901558]
• 8: Graeme Hirst , Alexander Budanitsky, Correcting real-word spelling errors by restoring lexical cohesion, Natural Language Engineering, v.11 n.1, p.87-111, March 2005 [doi>10.1017/S1351324904003560]
• 9: Paul B. Kantor , Ellen M. Voorhees, The TREC-5 Confusion Track: Comparing Retrieval Methods for Scanned Text, Information Retrieval, v.2 n.2-3, p.165-176, May 2000 [doi>10.1023/A:1009902609570]
• 10: Karen Kukich, Technique for automatically correcting words in text, ACM Computing Surveys (CSUR), v.24 n.4, p.377-439, Dec. 1992 [doi>10.1145/146370.146380]
• 11: Daniel Lopresti, Performance evaluation for text processing of noisy inputs, Proceedings of the 2005 ACM symposium on Applied computing, March 13-17, 2005, Santa Fe, New Mexico [doi>10.1145/1066677.1066851]
• 12: Eric Mays , Fred J. Damerau , Robert L. Mercer, Context based spelling correction, Information Processing and Management: an International Journal, v.27 n.5, p.517-522, 1991 [doi>10.1016/0306-4573(91)90066-U]
• 13: S. Mihov, P. Mitankin, A. Gotscharek, U. Reffle, K. U. Schulz, and C. Ringlstetter. Using automated error profiling of texts for improved selection of correction candidates for garbled tokens. In Australian Conference on Artificial Intelligence (AI2007), volume 4830 of Lecture Notes in Computer Science, pages 456--465, 2007.
• 14: Roger Mitton, Spelling checkers,spelling correctors and the misspellings of poor spellers, Information Processing and Management: an International Journal, v.23 n.5, p.495-505, Sept. 1987 [doi>10.1016/0306-4573(87)90116-6]
• 15: M. Reynaert. All, and only, the errors: more complete and consistent spelling and ocr-error correction evaluation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), 2008.
• 16: Ch. Ringlstetter , U. Reffle , A. Gotscharek , K. Schulz, Deriving Symbol Dependent Edit Weights for Text Correction_The Use of Error Dictionaries, Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol 2, p.639-643, September 23-26, 2007
• 17: Kazem Taghva , Julie Borsack , Allen Condit, Effects of OCR errors on ranking and feedback using the vector space model, Information Processing and Management: an International Journal, v.32 n.3, p.317-327, May 1996 [doi>10.1016/0306-4573(95)00058-5]
• 18: Kazem Taghva , Julie Borsack , Allen Condit , Srinivas Erva, The effects of noisy data on text retrieval, Journal of the American Society for Information Science, v.45 n.1, p.50-58, Jan. 1994 [doi>10.1002/(SICI)1097-4571(199401)45:1<50::AID-ASI6>3.0.CO;2-B]
• 19: L. A. Wilcox-O'Hearn, G. Hirst, and A. Budanitsky. Real-word spelling correction with trigrams: A reconsideration of the Mays, Damerau, and Mercer model. In Proc., 9th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2008), pages 605--616, Haifa, 2008.
• 20: David Yarowsky, Decision lists for lexical ambiguity resolution: application to accent restoration in Spanish and French, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.88-95, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981745]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.1 Document and Text Editing

Additional Classification:
  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation

Keywords:
error correction, error dictionaries, false friends