Capturing a taxonomy of failures during automatic interpretation of questions posed in natural language

ABSTRACT

An important problem in artificial intelligence is capturing, from natural language, formal representationsallthat can be used by a reasoner to compute an answer. Many researchers have studied this problem by developing algorithms addressing specific phenomena in natural language interpretation, but few have studied (or cataloged) the types of failures associated with this problem. Knowledgeallof these failures can help researchers by providing a roadallmap of open research problems and help practitioners by providing a checklist of issues to address in order to build systems that can achieve good performance on this problem.allIn this paper, we present a study -- conducted in the context of the Halo Project -- cataloging the types of failures that occur when capturing knowledge from naturallanguage. We identified the categories of failures by examining a corpus of questions posed byallnaive usersallto a knowledge based question answering system and empirically demonstrated the generality of ourallcategorizations. We also describe available technologies that can address some of the failures we have identified.

REFERENCES

• 1: K. Barker, B. Agashe, S. Chaw, J. Fan, M. Glass, J. Hobbs, E. Hovy, D. Israel, D.S. Kim, R. Mulkar, S. Patwardhan, B. Porter, D. Tecuci, and P. Yeh. Learning by Reading: A Prototype System, Performance Baseline and Lessons Learned. In Proceedings of Twenty-Second National Conference on Artificial Intelligence, 2007.
• 2: S. Chaw, K. Barker, B. Porter, and P. Yeh. An Ontology-Independent Problem Solver. Technical report, University of Texas at Austin, May 2007.
• 3: Peter Clark , Vinay Chaudhri , Sunil Mishra , Jérôme Thoméré , Ken Barker , Bruce Porter, Enabling domain experts to convey questions to a machine: a modified, template-based approach, Proceedings of the 2nd international conference on Knowledge capture, October 23-25, 2003, Sanibel Island, FL, USA [doi>10.1145/945645.945651]
• 4: Peter Clark , Shaw-Yi Chaw , Ken Barker , Vinay Chaudhri , Philip Harrison , James Fan , Bonnie John , Bruce Porter , Aaron Spaulding , John Thompson , Peter Yeh, Capturing and answering questions posed to a knowledge-based system, Proceedings of the 4th international conference on Knowledge capture, October 28-31, 2007, Whistler, BC, Canada [doi>10.1145/1298406.1298419]
• 5: P.R. Cohen, R. Schrag, E.K. Jones, A. Pease, A. Lin, B. Starr, D. Gunning, and M. Burke. The DARPA High-Performance Knowledge Bases project. AI Magazine, 19(4):25--49, 1998.
• 6: J. Fan, K. Barker, and B. Porter. The knowledge required to interpret noun compounds. In Proceedings of Eighteenth International Joint Conference on Artificial Intelligence, 2003.
• 7: James Fan , Ken Barker , Bruce Porter, Indirect anaphora resolution as semantic path search, Proceedings of the 3rd international conference on Knowledge capture, October 02-05, 2005, Banff, Alberta, Canada [doi>10.1145/1088622.1088650]
• 8: J. Fan and B. Porter. Interpret loosely encoded questions. In Proceedings of Nineteenth National Conference on Artificial Intelligence. AAAI Press, 2004.
• 9: N. Friedland, P. Allen, P. Matthews, M. Witbrock, D. Baxter, J. Curtis, B. Shepard, P. Miraglia, J. Angele, S. Staab, E. Moench, H. Oppermann, D. Wenke, D. Israel, V. Chaudhri, B. Porter, K. Barker, J. Fan, S. Chaw, P. Yeh, D. Tecuci, and P. Clark. Project Halo: Towards a Digital Aristotle. AI Magazine, 2004.
• 10: N. Friedland, P. Allen, M. Witbrock, J. Angele, S. Staab, D. Israel, V. Chaudhri, B. Porter, K. Barker, and P. Clark. Towards a quantitative, platform-independent analysis of knowledge systems. In Proceedings of Ninth International Conference on the Principles of Knowledge Representation and Reasoning, 2004.
• 11: Daniel Gildea , Daniel Jurafsky, Automatic labeling of semantic roles, Computational Linguistics, v.28 n.3, p.245-288, September 2002 [doi>10.1162/089120102760275983]
• 12: J.R. Landis and G.G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33:159--174, 1977.
• 13: Rada Mihalcea , Paul Tarau , Elizabeth Figa, PageRank on semantic networks, with application to word sense disambiguation, Proceedings of the 20th international conference on Computational Linguistics, p.1126-es, August 23-27, 2004, Geneva, Switzerland [doi>10.3115/1220355.1220517]
• 14: R. Mitkov. Anaphora Resolution. Longman, London, 2002.
• 15: V. Nastase, J. Sayyad-Shirabad, M. Sokolova, and S. Szpakowicz. Learning noun-modifer semantic relations with corpus--based and wordnet-based features. In Proceedings of the Twenty-first National Conference on Artificial Intelligence (AAAI '07), 2006.
• 16: R. Schrag, M. Pool, V. Chaudhri, R.C. Kahlert, J. Powers, P. Cohen, JFitzgerald, and M.S. Experimental evaluation of subject matter expert-oriented. Technical report, Information Extraction and Transport, Inc., August 2002.
• 17: Proceedings of the 2002 PerMIS Workshop, August 13--15, 2002, NIST Special Publication 990, pp. 272--279.
• 18: J. F. Sowa, Conceptual structures: information processing in mind and machine, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1984
• 19: D. Tecuci and B. Porter. A Generic Memory Module for Events. In Proceedings to the FLAIRS20 Conference, Key West, FL, 2007.
• 20: Teknowledge Corporation. Rapid Knowledge Formation project, 2002. http://reliant.teknowledge.com/RKF.
• 21: Vulcan Inc. Project Halo, 2003. http://projecthalo.com.
• 22: Peter Z. Yeh , Bruce Porter , Ken Barker, Using transformations to improve semantic matching, Proceedings of the 2nd international conference on Knowledge capture, October 23-25, 2003, Sanibel Island, FL, USA [doi>10.1145/945645.945671]
• 23: Peter Z. Yeh , Bruce Porter , Ken Barker, Matching utterances to rich knowledge structures to acquire a model of the speaker's goal, Proceedings of the 3rd international conference on Knowledge capture, October 02-05, 2005, Banff, Alberta, Canada [doi>10.1145/1088622.1088646]
• 24: P. Yeh, B. Porter, and K. Barker. A unified knowledge based approach for sense disambiguation and semantic role labeling. In Proceedings of Twenty-First National Conference on Artificial Intelligence. AAAI Press, 2006.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.0 General

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Human Factors

Keywords:
NLP, controlled languages, knowledge based systems, ontology, question answering