Selecting indexing strings using adaptation

ABSTRACT

It is not easy to tokenize agglutinative languages like Japanese and Chinese into words. Many IR systems start with a dictionary-based morphology program like ChaSen [4]. Unfortunately, dictionaries cannot cover all possible words; unknown words such as proper nouns are important for IR. This paper proposes a statistical dictionary-free method for selecting index strings based on recent work on adaptive language modeling.

REFERENCES

• 1: Kenneth W. Church (2000), Empirical Estimates of Adaptation: The chance of Two Noriegas is closer to p/2 than p2, Coling, pp. 173--179.
• 2: NTCIR Project, http://research.nii.ac.jp/ntcir/
• 3: Kyoji Umemura , Kenneth W. Church, Empirical term weighting and expansion frequency, Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, p.117-123, October 07-08, 2000, Hong Kong [doi>10.3115/1117794.1117809]
• 4: Yuji Matsumoto, Akira Kitauchi, Tatsuo Yamashita, Yoshitaka Hirano, Osamu Imaichi, and Tomoaki Imamura (1997), Japanese Morphological analysis System ChaSen Manual, NAIST Technical Report NAIST-IS-TR97007, http://cactus.aist-nara.ac.jp/lab/nlt/chasen.html

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

General Terms:
Algorithms

Keywords:
adaptation, agglutinative language, word segmentation