Unsupervised question answering data acquisition from local corpora

ABSTRACT

Data-driven approaches in question answering (QA) are increasingly common. Since availability of training data for such approaches is very limited, we propose an unsupervised algorithm that generates high quality question-answer pairs from local corpora. The algorithm is ontology independent, requiring very small seed data as its starting point. Two alternating views of the data make learning possible: 1) question types are viewed as relations between entities and 2) question types are described by their corresponding question-answer pairs. These two aspects of the data allow us to construct an unsupervised algorithm that acquires high precision question-answer pairs. We show the quality of the acquired data for different question types and perform a task-based evaluation. With each iteration, pairs acquired by the unsupervised algorithm are used as training data to a simple QA system. Performance increases with the number of question-answer pairs acquired confirming the robustness of the unsupervised algorithm. We introduce the notion of <i>semantic drift</i> and show that it is a desirable quality in training data for question answering systems.

REFERENCES

• 1: C. Clarke, G. Cormack, G. Kemkes, M. Laszlo, T. Lynam, E. Terra, and P. Tilker. Statistical selection of exact answers. Text Retrieval Conference (TREC), 2003.
• 2: Charles L. A. Clarke , Gordon V. Cormack , Thomas R. Lynam, Exploiting redundancy in question answering, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.358-365, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384024]
• 3: M. Collins and Y. Singer. Unsupervised models for named entity classification. Conference on Empirical Methods in Natural Language Processing (EMNLP)/VLC, 1999.
• 4: Susan Dumais , Michele Banko , Eric Brill , Jimmy Lin , Andrew Ng, Web question answering: is more always better?, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564428]
• 5: Abdessamad Echihabi , Daniel Marcu, A noisy-channel approach to question answering, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.16-23, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075099]
• 6: Michael Fleischman , Eduard Hovy , Abdessamad Echihabi, Offline strategies for online question answering: answering questions before they are asked, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.1-7, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075097]
• 7: Roxana Girju , Adriana Badulescu , Dan Moldovan, Learning semantic constraints for the automatic discovery of part-whole relations, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, p.1-8, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073445.1073456]
• 8: U. Hermjakob, E. Hovy, and C. Lin. Knowledge-based question answering. Text Retrieval Conference (TREC), 2000.
• 9: E. Hovy, L. Gerber, U. Hermjakob, M. Junk, and C. Lin. Question answering in webclopedia. Text Retrieval Conference (TREC), 2000.
• 10: Eduard Hovy , Ulf Hermjakob , Chin-Yew Lin , Deepak Ravichandran, Using knowledge to facilitate factoid answer pinpointing, Proceedings of the 19th international conference on Computational linguistics, p.1-7, August 24-September 01, 2002, Taipei, Taiwan [doi>10.3115/1072228.1072270]
• 11: Lita and J. Carbonell. Instance-based question answering: A data-driven approach. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2004.
• 12: Lita, W. Hunt, and E. Nyberg. Resource analysis for question answering. Association for Computational Linguistics Conference (ACL), 2004.
• 13: B. Magnini, S. Romagnoli, A. Vallin, J. Herrera, A. Penas, V. Peiado, F. Verdejo, and M. de Rijke. The multiple language question answering track at cross-lingual evaluation forum (clef) 2003. Cross-Lingual Evaluation Forum (CLEF), 2003.
• 14: G. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. Miller. Five papers on wordnet. International Journal of Lexicography, 1990.
• 15: D. Moldovan, D. Clark, S. Harabagiu, and S. Maiorano. Cogex: A logic prover for question answering. Association for Computational Linguistics Conference (ACL), 2003.
• 16: Dan Moldovan , Sanda Harabagiu , Marius Pasca , Rada Mihalcea , Roxana Girju , Richard Goodrum , Vasile Rus, The structure and performance of an open-domain question answering system, Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, p.563-570, October 03-06, 2000, Hong Kong [doi>10.3115/1075218.1075289]
• 17: E. Nyberg, T. Mitamura, J. Callan, J. Carbonell, R. Frederking, K. Collins-Thompson, L. Hiyakumoto, Y. Huang, C. Huttenhower, S. Judy, J. Ko, A. Kupsc, L. V. Lita, V. Pedro, D. Svoboda, and B. V. Durme. The javelin question-answering system at trec 2003: A multi strategy approach with dynamic planning. Text Retrieval Conference (TREC), 2003.
• 18: Deepak Ravichandran , Eduard Hovy, Learning surface text patterns for a Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073092]
• 19: Deepak Ravichandran , Abraham Ittycheriah , Salim Roukos, Automatic derivation of surface text patterns for a maximum entropy based question answering system, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL 2003--short papers, p.85-87, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073483.1073512]
• 20: Michael Thelen , Ellen Riloff, A bootstrapping method for learning semantic lexicons using extraction pattern contexts, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.214-221, July 06, 2002 [doi>10.3115/1118693.1118721]
• 21: E. Voorhees. Overview of the text retrieval conference (trec) 2003 question answering track. Text Retrieval Conference (TREC), 2003.
• 22: David Yarowsky, Decision lists for lexical ambiguity resolution: application to accent restoration in Spanish and French, Proceedings of the 32nd annual meeting on Association for Computational Linguistics, p.88-95, June 27-30, 1994, Las Cruces, New Mexico [doi>10.3115/981732.981745]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Algorithms, Experimentation

Keywords:
data acquisition, question answering, semantic drift, unsupervised learning