Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Stemming and lemmatization were compared in the clustering of Finnish text documents. Since Finnish is a highly inflectional and agglutinative language, we hypothesized that lemmatization, involving splitting of the compound words, would be more appropriate normalization approach than the straightforward stemming. The relevance of the documents were evaluated with a four-point relevance assessment scale, which was collapsed into binary one by considering all the relevant and only the highly relevant documents relevant, respectively. Experiments with four hierarchical clustering methods supported the hypothesis. The stringent relevance scale showed that lemmatization allowed the single and complete linkage methods to recover especially the highly relevant documents better than stemming. In comparison with stemming, lemmatization together with the average linkage and Ward's methods produced higher precision. We conclude that lemmatization is a better word normalization method than stemming, when Finnish text documents are clustered for information retrieval.

REFERENCES

• 1: Riitta Alkula, From Plain Character Strings to Meaningful Words: Producing Better Full Text Databases for Inflectional and Compounding Languages with Morphological Analysis Software, Information Retrieval, v.4 n.3-4, p.195-208, September-December 2001 [doi>10.1023/A:1011942104443]
• 2: Robert Krovetz, Viewing morphology as an inference process, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.191-202, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160718]
• 3: Pirkola, A. Morphological typology of languages for information retrieval. Journal of Documentation, 57, 3 (2001), 330--348.
• 4: Harman D. How effective is suffixing? Journal of the American Society for Information Science, 42, 1 (1991), 7--15.
• 5: David A. Hull, Stemming algorithms: a case study for detailed evaluation, Journal of the American Society for Information Science, v.47 n.1, p.70-84, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<70::AID-ASI7>3.3.CO;2-Q]
• 6: Popovic, M., and Willett, P. The effectiveness of stemming for natural-language access to Slovene textual data. Journal of the American Society for Information Science, 43, 1 (1992), 384--390.
• 7: Jacques Savoy, A stemming procedure and stopword list for general French corpora, Journal of the American Society for Information Science, v.50 n.10, p.944-952, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:10<944::AID-ASI9>3.3.CO;2-H]
• 8: Kalamboukis, T. Z. Suffix stripping with modern Greek. Program, 29, 3 (1995), 313--321.
• 9: Hani Abu-Salem , Mahmoud Al-Omari , Martha W. Evens, Stemming methodologies over individual query words for an Arabic information retrieval system, Journal of the American Society for Information Science, v.50 n.6, p.524-529, May 1, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:6<524::AID-ASI7>3.3.CO;2-D]
• 10: Rosell, M. Improving clustering of Swedish newspaper articles using stemming and compound splitting. In Fourteenth Nordic Conference on Computational Linguistics (NoDaLiDa 2003) (Reykjavik, Island, May 30-31, 2003). http://www.nada.kth.se/~rosell/publications/papers/improvingClustering03.pdf
• 11: Matthews, P. H. The Concise Oxford Dictionary of Linguistics. Oxford University Press, Oxford - New York, NY, 1997.
• 12: Peter Willett, Recent trends in hierarchic document clustering: a critical review, Information Processing and Management: an International Journal, v.24 n.5, p.577-597, 1988 [doi>10.1016/0306-4573(88)90027-1]
• 13: P. Willett , A. El-Hamdouchi, Comparison of hierarchic agglomerative clustering methods for document retrieval, The Computer Journal, v.32 n.3, p.220-227, June 1989 [doi>10.1093/comjnl/32.3.220]
• 14: Martin Nilsson, Hierarchical Clustering Using Non-Greedy Principal Direction Divisive Partitioning, Information Retrieval, v.5 n.4, p.311-321, October 2002 [doi>10.1023/A:1020443310743]
• 15: Kekäläinen, J. The Effects of Query Complexity, Expansion and Structure on Retrieval Performance in Probabilistic Text Retrieval. Ph.D. Thesis, University of Tampere, 1999. Acta Universitatis Tamperensis, vol. 678.
• 16: Sormunen, E. A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases. Ph.D. Thesis, University of Tampere, 2000. Acta Universitatis Tamperensis, vol. 748.
• 17: Jaana Kekäläinen , Kalervo Järvelin, Using graded relevance assessments in IR evaluation, Journal of the American Society for Information Science and Technology, v.53 n.13, p.1120-1129, November 2002 [doi>10.1002/asi.10137]
• 18: Karlsson, F. Finnish grammar. WSOY, Porvoo, 1987.
• 19: Koskenniemi, K. An application of the two-level model to Finnish. In Computational morphosyntax: Report on research 1981-84. Publications 13, University of Helsinki, Department of General Linguistics, Helsinki, 1985, 19--41.
• 20: Koskenniemi, K. Two-level morphology: A general computational model for word-form recognition and production. Publications 11, University of Helsinki, Department of General Linguistics, Helsinki, 1983.
• 21: Porter, M. F. Snowball: A language for stemming algorithms, 2001. http://snowball.tartarus.org/
• 22: Jolliffe, I. T. Principal Components Analysis. Springer-Verlag, New York, 1986.
• 23: Korenius, T., Laurikkala, J., and Juhola, M. On applying the principal components analysis and cosine similarity for information retrieval. A manuscript submitted to Information Processing & Management.
• 24: Tuomo Korenius , Jorma Laurikkala , Martti Juhola , Kalervo Järvelin, Hierarchical clustering of a Finnish newspaper article collection with graded relevance assessments, Information Retrieval, v.9 n.1, p.33-53, January 2006 [doi>10.1007/s10791-005-5720-6]
• 25: Edie Rasmussen, Clustering algorithms, Information retrieval: data structures and algorithms, Prentice-Hall, Inc., Upper Saddle River, NJ, 1992
• 26: Everitt, B. S., Landau, S., and Leese, M. Cluster Analysis. Arnold, London, 2001.
• 27: The Math Works Inc. Statistics Toolbox User's Guide. The Math Works Inc., Natick, 2002.
• 28: Pett, M. A. Nonparametric Statistics for Health Care Research. Sage Publications, Thousand Oaks, CA, 1997.
• 29: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

General Terms:
Algorithms, Experimentation, Standardization

Keywords:
clustering, lemmatization, normalization, stemming