Document retrieval for question answering: a quantitative evaluation of text preprocessing

ABSTRACT

Question Answering (QA) has been an area of interest for researchers, in part motivated by the international QA evaluation forums, namely the Text REtrieval Conference (TREC), and more recently, the Cross Language Evaluation Forum (CLEF) through QA@CLEF, that since 2004 includes the Portuguese language. In these forums, a collection of written documents is provided, as well as a set of questions, which are to be answered by the participating systems. Each system is evaluated by its capacity to answer the questions, as a whole, and there are relatively few results published that focus on the performance of its different components and their influence on the overall system performance. That is the case of the Information Retrieval (IR) component, which is broadly used in QA systems.

Our work concentrates on the different options of preprocessing Portuguese text before feeding it to the IR component, evaluating their impact on the IR performance in the specific context of QA, so that we can make a sustained choice of which options to choose. From this work we conclude the clear advantage of the basic preprocessing techniques: case folding and removal of punctuation marks. For the other techniques considered, stop word removal enhanced the performance of the IR system but that was not the case as far as Stemming and Lemmatization are concerned.

REFERENCES

• 1: Alves, M. A. Engenharia do Léxico Computacional: princípios, tecnologia e o caso das palavras compostas. Mestrado emEngenharia Informática. Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa, (20 Feb. 2002). www.liacc.up.pt/~maa/elc.html
• 2: Bilotti, M.W., Katz, B. and Lin, J. What Works Better for Question Answering: Stemming or Morphological Query Expansion? ACM SIGIR'04 Workshop Information Retrieval for QA, (Jul. 2004).
• 3: Christopher Fox, A stop list for general text, ACM SIGIR Forum, v.24 n.1-2, p.19-21, Fall 89/Winter 90 [doi>10.1145/378881.378888]
• 4: Roberts, I., and Gaizauskas, R. Evaluating Passage Retrieval Approaches for Question Answering. Lecture Notes in Computer Science, Book: Advances in Information Retrieval, Volume 2997, (Mar. 2004), 72--84.
• 5: Santos, D. and Rocha,P. CHAVE: topics and questions on the Portuguese participation in CLEF. In C. Peters and F. Borri, editors, Cross Language Evaluation Forum: Working Notes for the CLEF 2004 Workshop, Bath, UK, (15--17 September 2004) Pg. 639--648
• 6: Stefanie Tellex , Boris Katz , Jimmy Lin , Aaron Fernandes , Gregory Marton, Quantitative evaluation of passage retrieval algorithms for question answering, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860445]
• 7: Ellen M. Voorhees , Dawn M. Tice, Building a question answering test collection, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.200-207, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345577]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Experimentation, Measurement, Performance

Keywords:
information retrieval, question answering