Management of keyword variation with frequency based generation of word forms in IR

ABSTRACT

This paper presents a new management method for morphological variation of keywords. The method is called FCG, Frequent Case Generation. It is based on the skewed distributions of word forms in natural languages and is suitable for languages that have either fair amount of morphological variation or are morphologically very rich. The proposed method has been evaluated so far with four languages, Finnish, Swedish, German and Russian, which show varying degrees of morphological complexity.

REFERENCES

• 1: Baayen, R. H. Statistical Models for Word Frequency Distribution. Computers and the Humanities 26 (1993): 347--363.
• 2: Baayen, R. H. Word Frequency Distributions. Kluwer Academic Publishers, Dordrecht Boston London, 2001.
• 3: Karlsson, F. Frequency Considerations in Morphology. Zeitsschrift fr Phonetik, Sprachwissenschaft und Kommunikationsforschung 39 (1986): 19--28.
• 4: Karlsson, F. Defectivity. In: Booij G. et al. (eds.): Morphology. An International Handbook on Inflection and Word-Formation. Volume 1. Walter de Gruyter, Berlin, 2000, 647--654.
• 5: Kettunen, K. and Airio, E. Is a morphologically complex language really that complex in full-text retrieval? In T. Salakoski et al. (Eds.): Advances in Natural Language Processing, LNAI 4139. Springer-Verlag Berlin Heidelberg, 2006, 411--422.
• 6: Kimmo Kettunen , Eija Airio , Kalervo Järvelin, Restricted inflectional form generation in management of morphological keyword variation, Information Retrieval, v.10 n.4-5, p.415-444, October 2007 [doi>10.1007/s10791-007-9030-z]
• 7: Kosti , A., Markovi , T. and Baucal, A. Inflectional Morphology and Word Meaning: Orthogonal or Co-implicative Cognitive Domains. In: Baayen, R.H. and Schreuder R. (eds.): Morphological Structure in Language Processing. Trends in Linguistics, Studies and Monographs 151. Mouton de Gruyter, Berlin, 2003, 1--43.
• 8: Perebeynoss, V. and Khidekel, S. Frequency of Language Units as a Reflection of Their Systemic and Functional Properties. Journal of Quantitative Linguistics 11 (2004): 3--25.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Design, Experimentation, Languages, Measurement, Performance

Keywords:
evaluation, management of morphological variation, monolingual information retrieval, word form generation