Corpus-based stemming using cooccurrence of word variants

ABSTRACT

Stemming is used in many information retrieval (IR) systems to reduce variant word forms to common roots. It is one of the simplest applications of natural-language processing to IR and is one of the most effective in terms of user acceptance and consistency, though small retrieval improvements. Current stemming techniques do not, however, reflect the language use in specific corpora, and this can lead to occasional serious retrieval failures. We propose a technique for using corpus-based word variant cooccurrence statistics to modify or create a stemmer. The experimental results generated using English newspaper and legal text and Spanish text demonstrate the viability of this technique and its advantages relative to conventional approaches that only employ morphological rules.

REFERENCES

• 1: BROGLIO, J., CALLAN, J. P., AND CROFT, W. 1994. An overview of the INQUERY system as used for the TIPSTER project. In Proceedings of the TIPSTER Workshop. Morgan-Kaufmann, San Mateo, Calif., 47-67.
• 2: BROGLIO, J., CALLAN, J. P., CROFT, W. B., AND NACHBAR, D.W. 1995. Document retrieval and routing using the INQUERY system. In Proceedings of the 3rd Text REtrieval Conference (TREC-3), D. Harman, Ed. NIST Special Publication 500-225, 22-29.
• 3: Kenneth Ward Church , Patrick Hanks, Word association norms, mutual information, and lexicography, Proceedings of the 27th annual meeting on Association for Computational Linguistics, p.76-83, June 26-29, 1989, Vancouver, British Columbia, Canada [doi>10.3115/981623.981633]
• 4: CROFT, W. B. AND XU, J. 1995. Corpus-specific stemming using word form co-occurrence. In the 4th Annual Symposium on Document Analysis and Information Retrieval. 147-159.
• 5: HARMAN, D. 1991. How effective is suffixing? J. Am. Soc. Inf. Sci. 42, 1, 7-15.
• 6: HARMAN, D. 1995. Overview of the third text REtrieval conference (TREC-3). In Proceedings of the 3rd Text REtrieval Conference (TREC-3), D. Harman, Ed. NIST Special Publication 500-225, 1-20.
• 7: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 8: David A. Hull, Stemming algorithms: a case study for detailed evaluation, Journal of the American Society for Information Science, v.47 n.1, p.70-84, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<70::AID-ASI7>3.3.CO;2-Q]
• 9: JING, Y. AND CROFT, W. 1994. An association thesaurus for information retrieval. In Proceedings of RIAO 94. 146-160.
• 10: Wessel Kraaij , Renée Pohlmann, Viewing stemming as recall enhancement, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.40-48, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243209]
• 11: Robert Krovetz, Viewing morphology as an inference process, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.191-202, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160718]
• 12: PoPovIc, M. AND WILLETT, P. 1992. The effectiveness of stemming for natural-language access to Slovene textual data. J. Am. Soc. Inf. Sci. 43, 5, 384-390.
• 13: PORTER, M. 1980. An algorithm for suffix stripping. Program 14, 3, 130-137.
• 14: Gerald Salton, Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988
• 15: SPARCK JONES, K. 1971. Automatic Keyword Classification for Information Retrieval. Archon Books, Hamden, Conn.
• 16: Howard Turtle, Natural language vs. Boolean query evaluation: a comparison of retrieval performance, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.212-220, July 03-06, 1994, Dublin, Ireland
• 17: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 18: Ellen M. Voorhees, Query expansion using lexical-semantic relations, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.61-69, July 03-06, 1994, Dublin, Ireland

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing
      H.3.3 Information Search and Retrieval
          Subjects: Search process; Query formulation

General Terms:
Algorithms, Experimentation, Performance

Keywords:
class refinement, cooccurence, corpus analysis, information retrieval, n-gram, stemming