Quantitative evaluation of passage retrieval algorithms for question answering

ABSTRACT

Passage retrieval is an important component common to many question answering systems. Because most evaluations of question answering systems focus on end-to-end performance, comparison of common components becomes difficult. To address this shortcoming, we present a quantitative evaluation of various passage retrieval algorithms for question answering, implemented in a framework called Pauchok. We present three important findings: Boolean querying schemes perform well in the question answering task. The performance differences between various passage retrieval algorithms vary with the choice of document retriever, which suggests significant interactions between document retrieval and passage retrieval. The best algorithms in our evaluation employ density-based measures for scoring query terms. Our results reveal future directions for passage retrieval and question answering.

REFERENCES

• 1: James P. Callan, Passage-level evidence in document retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.302-310, July 03-06, 1994, Dublin, Ireland
• 2: C. Clarke, G. Cormack, D. Kisman, and T. Lynam. Question answering by passage selection (Multitext experiments for TREC-9). In Proceedings of the Ninth Text REtrieval Conference (TREC-9)2000.
• 3: Charles L. A. Clarke , Gordon V. Cormack , Elizabeth A. Tudhope, Relevance ranking for one to three term queries, Information Processing and Management: an International Journal, v.36 n.2, p.291-311, Jan.1.2000 [doi>10.1016/S0306-4573(99)00017-5]
• 4: William S. Cooper, Getting beyond Boole, Information Processing and Management: an International Journal, v.24 n.3, p.243-248, May 1988 [doi>10.1016/0306-4573(88)90091-X]
• 5: S. Harabagiu, D. Moldovan, M. Pasca, M. Surdeanu, R. Mihalcea, R. Girju, V. Rus, F. Lacatusu, P. Morarescu, and R. Bunescu. Answering complex, list, and context questions with LCC's Question-Answering Server. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 6: Sanda M. Harabagiu , Marius A. Paşca , Steven J. Maiorano, Experiments with open-domain textual Question Answering, Proceedings of the 18th conference on Computational linguistics, p.292-298, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990863]
• 7: L. Hirschman , R. Gaizauskas, Natural language question answering: the view from here, Natural Language Engineering, v.7 n.4, p.275-300, December 2001 [doi>10.1017/S1351324901002807]
• 8: E. Hovy, U. Hermjakob, and C.-Y. Lin. The use of external knowledge in factoid QA. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001) 2001.
• 9: A. Ittycheriah, M. Franz, and S. Roukos. IBM's statistical question answering system-TREC-10.In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 10: A. Ittycheriah, M. Franz, W.-J. Zhu, and A. Ratnaparkhi. IBM's statistical question answering system. In Proceedings of the 9th Text REtrieval Conference (TREC-9)2000.
• 11: B. Katz and J. Lin. Selectively using relations to improve precision in question answering. In Proceedings of the EACL-2003 Workshop on Natural Language Processing for Question Answering 2003.
• 12: G. G. Lee, J. Seo, S. Lee, H. Jung, B.-H. Cho, C. Lee, B.-K. Kwak, J. Cha, D. Kim, J. An, H. Kim, and K. Kim. SiteQ: Engineering high performance QA system using lexico-semantic pattern matching and shallow NLP. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 13: Marc Light , Gideon S. Mann , Ellen Riloff , Eric Breck, Analyses for elucidating current question answering technology, Natural Language Engineering, v.7 n.4, p.325-342, December 2001 [doi>10.1017/S1351324901002819]
• 14: Jimmy Lin , Dennis Quan , Vineet Sinha , Karun Bakshi , David Huynh , Boris Katz , David R. Karger, The role of context in question answering systems, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.766119]
• 15: Fernando Llopis , José Luis Vicedo González, IR-n: A Passage Retrieval System at CLEF-2001, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.244-252, September 03-04, 2001
• 16: Dan Moldovan , Marius Paşca , Sanda Harabagiu , Mihai Surdeanu, Performance issues and error analysis in an open-domain Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073091]
• 17: S. E. Robertson, S. Walker, M. Hancock-Beaulieu, M. Gatford, and A. Payne. Okapi at TREC-4. In Proceedings of the 4th Text REtrieval Conference (TREC-4)1995.
• 18: S. E. Robertson, S. Walker, S. Jones, M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Proceedings of the 3rd Text REtrieval Conference (TREC-3)1994.
• 19: Gerard Salton , J. Allan , Chris Buckley, Approaches to passage retrieval in full text information systems, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.49-58, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160693]
• 20: M. M. Soubbotin and S. M. Soubbotin. Patterns of potential answer expressions as clues to the right answers. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 21: R. Srihari and W. Li. Information extraction supported question answering. In Proceedings of the Eighth Text REtrieval Conference (TREC-8)1999.
• 22: J. L. Vicedo and A. Ferrandez. University of Alicante at TREC-10. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 23: E. M. Voorhees. Overview of the TREC 2001 question answering track. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001)2001.
• 24: E. M. Voorhees. Overview of the TREC 2002 question answering track. In Proceedings of the Eleventh Text REtrieval Conference (TREC 2002)2002.
• 25: E. M. Voorhees and D. M. Tice. Overview of the TREC-9 question answering track. In Proceedings of the Ninth Text REtrieval Conference (TREC-9)2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Design, Measurement, Performance, Standardization

Keywords:
passage retrieval, question answering