A novel statistical chinese language model and its application in pinyin-to-character conversion

ABSTRACT

In this paper, we present a novel Chinese language model, and study its applications, in particular in Chinese pinyin-to-character conversion. In the new model, each word is associated with supporting context constructed by mining the frequent sets of nearby phrases and their distances to the word. Such information was usually overlooked in previous n-gram model and its variants. We apply the model to Chinese pinyin-to-character conversion and find that it offers a better solution to Chinese input. The model has lower perplexity in our evaluation and higher prediction accuracy than the state-of-the-art n-gram Markov model for Chinese language.

REFERENCES

• 1: Intelligent Pinyin Input Method Editor Demo Website, http://www.cais.ntu.edu.sg/~jzhang/pinyin/index_en.html.
• 2: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, ACM SIGMOD Record, v.22 n.2, p.207-216, June 1, 1993
• 3: Adam L. Berger , Vincent J. Della Pietra , Stephen A. Della Pietra, A maximum entropy approach to natural language processing, Computational Linguistics, v.22 n.1, p.39-71, March 1996
• 4: H. Cheng, X. Yan, J. Han, and C. Hsu. Discriminative frequent pattern analysis for effective classification. In ICDE, pages 716--725, 2007.
• 5: Jianfeng Gao , Joshua Goodman , Mingjing Li , Kai-Fu Lee, Toward a unified approach to statistical language modeling for Chinese, ACM Transactions on Asian Language Information Processing (TALIP), v.1 n.1, p.3-33, March 2002 [doi>10.1145/595576.595578]
• 6: Xiaoqiang Luo , Salim Roukos, An iterative algorithm to build Chinese language models, Proceedings of the 34th annual meeting on Association for Computational Linguistics, p.139-143, June 24-27, 1996, Santa Cruz, California [doi>10.3115/981863.981882]
• 7: David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Proceedings of the 33rd annual meeting on Association for Computational Linguistics, p.189-196, June 26-30, 1995, Cambridge, Massachusetts [doi>10.3115/981658.981684]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language models

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
chinese language model, pinyin-to-character conversion, word-context support