Light stemming approaches for the French, Portuguese, German and Hungarian languages

ABSTRACT

This paper describes and evaluates various general stemming approaches for the French, Portuguese (Brazilian), German and Hungarian languages. Based on the CLEF test-collections, we demonstrate that light stemmers for the French, Portuguese and Hungarian languages perform well, and reasonably well for the German language. Variations in mean average precision among the different stemming approaches are also evaluated and sometimes they are found statistically significant.

REFERENCES

• 1: Paul Mcnamee , James Mayfield, Character N-Gram Tokenization for European Language Text Retrieval, Information Retrieval, v.7 n.1-2, p.73-97, January-April 2004 [doi>10.1023/B:INRT.0000009441.78971.be]
• 2: Lovins, J. B. Development of a stemming algorithm. Mechanical Translation and Computational Linguistics, 11(1), 1968, 22--31.
• 3: Porter, M. F. An algorithm for suffix stripping. Program, 14(3), 1980, 130--137.
• 4: Robert Krovetz, Viewing morphology as an inference process, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.191-202, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160718]
• 5: Jinxi Xu , W. Bruce Croft, Corpus-based stemming using cooccurrence of word variants, ACM Transactions on Information Systems (TOIS), v.16 n.1, p.61-81, Jan. 1998 [doi>10.1145/267954.267957]
• 6: Savoy, J. Stemming of French words based on grammatical category. JASIS, 44(1), 1993, 1--9.
• 7: Tuomo Korenius , Jorma Laurikkala , Kalervo Järvelin , Martti Juhola, Stemming and lemmatization in the clustering of finnish text documents, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA [doi>10.1145/1031171.1031285]
• 8: Harman, D. How effective is suffixing? JASIS, 42(1), 1991, 7--15.
• 9: Di Nunzio, G. M., Ferro, N., Melucci, M., and Orio, N. Experiments to evaluate probabilistic models for automatic stemmer generation and query word translation. In Comparative Evaluation of Multilingual Information Access Systems. LNCS #3237, Springer, Berlin, 2004, 220--235.
• 10: Martin Braschler , Bärbel Ripplinger, How Effective is Stemming and Decompounding for German Text Retrieval?, Information Retrieval, v.7 n.3-4, p.291-316, September-December 2004 [doi>10.1023/B:INRT.0000011208.60754.a1]
• 11: Tomlinson, S. Lexical and algorithmic stemming compared for 9 European languages with Humminbird SearchServer#8482; at CLFF 2003. In Comparative Evaluation of Multilingual Information Access Systems. LNCS #3237, Springer-Verlag, Berlin, 2004, 286--300.
• 12: Kluck, M. The GIRT data in the evaluation of CLIR systems - from 1997 until 2003. In Comparative Evaluation of Multilingual Information Access Systems. LNCS #3237, Springer, Berlin, 2004, 376--390.
• 13: Savoy, J. Report on CLEF-2003 monolingual tracks. In Comparative Evaluation of Multilingual Information Access Systems. LNCS #3237, Springer, Berlin, 2004, 322--336.
• 14: Buckley, C., Singhal, A., Mitra, M., and Salton, G. New retrieval approaches using SMART. In Proceedings of TREC-4. Gaithersburg, MA, 1996, 25--48.
• 15: Singhal, A., Choi, J., Hindle, D., Lewis, D. D. & Pereira, F. (1999). AT&T at TREC-7. In Proceedings TREC-7, Gaithersburg, MA, 1999, 239--251.
• 16: S. E. Robertson , S. Walker , M. Beaulieu, Experimentation as a way of life: Okapi at TREC, Information Processing and Management: an International Journal, v.36 n.1, p.95-108, Jan. 2000 [doi>10.1016/S0306-4573(99)00046-1]
• 17: Gianni Amati , Cornelis Joost Van Rijsbergen, Probabilistic models of information retrieval based on measuring the divergence from randomness, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.357-389, October 2002 [doi>10.1145/582415.582416]
• 18: Jacques Savoy, Statistical inference in retrieval effectiveness evaluation, Information Processing and Management: an International Journal, v.33 n.4, p.495-512, July 1997 [doi>10.1016/S0306-4573(97)00027-7]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
      H.3.4 Systems and Software

General Terms:
Algorithms, Measurement, Performance

Keywords:
german, hungarian, natural language processing, portuguese, stemmer, stemming for french