Question answering passage retrieval using dependency relations

ABSTRACT

State-of-the-art question answering (QA) systems employ term-density ranking to retrieve answer passages. Such methods often retrieve incorrect passages as relationships among question terms are not considered. Previous studies attempted to address this problem by matching dependency relations between questions and answers. They used strict matching, which fails when semantically equivalent relationships are phrased differently. We propose fuzzy relation matching based on statistical models. We present two methods for learning relation mapping scores from past QA pairs: one based on mutual information and the other on expectation maximization. Experimental results show that our method significantly outperforms state-of-the-art density-based passage retrieval methods by up to 78% in mean reciprocal rank. Relation matching also brings about a 50% improvement in a system enhanced by query expansion.

REFERENCES

• 1: Y. Al-Onaizan, J. Curin, M. Jahr, K. Knight, J. Lafferty, D. Melamed, F. Och, D. Purdy, N. Smith, and D. Yarowsky, Statistical machine translation, Final Report, JHU Summer Workshop, 1999.
• 2: G. Attardi, A. Cisternino, F. Formica, M. Simi and A. Tommasi, PiQASso: Pisa Question Answering System, Proc. of TREC-2001, 2001, pp. 599--607.
• 3: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 4: Peter F. Brown , Vincent J. Della Pietra , Stephen A. Della Pietra , Robert L. Mercer, The mathematics of statistical machine translation: parameter estimation, Computational Linguistics, v.19 n.2, June 1993
• 5: H. Cui, K. Li, R. Sun, T.-S. Chua and M.-Y. Kan, National University of Singapore at the TREC-13 Question Answering Main Task, Proc. of TREC-13, 2004.
• 6: Abdessamad Echihabi , Daniel Marcu, A noisy-channel approach to question answering, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.16-23, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075099]
• 7: Jianfeng Gao , Jian-Yun Nie , Guangyuan Wu , Guihong Cao, Dependence language model for information retrieval, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009024]
• 8: S. Harabagiu, D. Moldovan, C. Clark, M. Bowden, J. Williams and J. Bensley, Answer Mining by Combining Extraction Techniques with Abductive Reasoning, Proc. of TREC-12, 2003, pp. 375--382.
• 9: David Hull, Using statistical testing in the evaluation of retrieval experiments, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.329-338, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160758]
• 10: A. Ittycheriah, M. Franz, and S. Roukos, IBM's statistical question answering system - TREC-10, Proc. of TREC-10, 2001.
• 11: Marcin Kaszkiel , Justin Zobel, Passage retrieval revisited, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.178-185, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 12: B. Katz and J. Lin, Selectively Using Relations to Improve Precision in Question Answering, Proc. of the EACL-2003 Workshop on Natural Language Processing for Question Answering, April 2003.
• 13: G. G. Lee, J. Seo, S. Lee, H. Jung, B.-H. Cho, C. Lee, B.-K. Kwak, J. Cha, D. Kim, J. An, H. Kim, and K. Kim, SiteQ: Engineering high performance QA system using lexico-semantic pattern matching and shallow NLP, Proc. of TREC-10, 2001, pp. 442--451.
• 14: Marc Light , Gideon S. Mann , Ellen Riloff , Eric Breck, Analyses for elucidating current question answering technology, Natural Language Engineering, v.7 n.4, p.325-342, December 2001 [doi>10.1017/S1351324901002819]
• 15: D. Lin, Dependency-based Evaluation of MINIPAR, Proc. of Workshop on the Evaluation of Parsing Systems, Granada, Spain, May, 1998.
• 16: J. Lin, D. Quan, V. Sinha, K. Bakshi, D. Huynh, B. Katz and D. R. Karger, What makes a good answer? The role of context in question answering, Proc. of the ninth IFIP TC13 International Conference on Human-Computer Interaction, 2003.
• 17: Dekang Lin , Patrick Pantel, Discovery of inference rules for question-answering, Natural Language Engineering, v.7 n.4, p.343-360, December 2001 [doi>10.1017/S1351324901002765]
• 18: Fei Song , W. Bruce Croft, A general language model for information retrieval, Proceedings of the eighth international conference on Information and knowledge management, p.316-321, November 02-06, 1999, Kansas City, Missouri, United States [doi>10.1145/319950.320022]
• 19: Stefanie Tellex , Boris Katz , Jimmy Lin , Aaron Fernandes , Gregory Marton, Quantitative evaluation of passage retrieval algorithms for question answering, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860445]
• 20: E.M. Voorhees, Overview of the TREC 2003 Question Answering Track, Proc. of TREC-12, pp. 54--68.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.1 Document and Text Editing

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
dependency parsing, passage retrieval, question answering