Evaluating expert-authored rules for military reasoning

ABSTRACT

Eliciting complex logical rules directly from logic-naive subject matter experts (SMEs) is a challenging knowledge capture task. We describe a large-scale experiment to evaluate tools designed to produce SME-authored rule bases. We assess the quality of the rule bases with respect to the: 1) performance on the addressed functional task (military course of action (COA) critiquing); and 2) intrinsic knowledge representation quality. In the course of this assessment, we note both strengths and weaknesses in the state of the art, and accordingly suggest some foci for future development in this important technology area.

REFERENCES

• 1: Barker, K., Blythe, J., Borchardt, G. Chaudhri, V., Clark, P., Cohen, P., Forbus, K., Gil, Y., Katz, B., Kim, J., King, G., Mishra, S., Murray, K., Otstott, C., Porter, B., Schrag, R., Uribe, T., Usher, J., and Yeh, P. "A Knowledge-Based Tool for Course of Action Analysis", to appear in the Fifteenth Conference on Innovative Applications of Artificial Intelligence (IAAI-03), 2003, Acapulco, Mexico.
• 2: Hans Chalupsky , Thomas A. Russ, WhyNot: debugging failed queries in large knowledge bases, Eighteenth national conference on Artificial intelligence, p.870-877, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 3: Cohen, P., Schrag, R., Jones, E., Pease, A., Lin, A., Starr, B., Gunning, D., and Burke, M. "The DARPA High-Performance Knowledge Bases Project." AI Magazine 19(4):25, 1998. Panther, J. G., Digital Communications, 3rd ed., Addison-Wesley, San Francisco, CA (1999).
• 4: DARPA, The Rapid Knowledge Formation Project, <http://reliant.teknowledge.com/RKF>, 2000.
• 5: Kenneth D. Forbus , Jeffrey Usher , Vernell Chapman, Sketching for military courses of action diagrams, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604059]
• 6: IET's RKF webpage. <http://www.iet.com/Projects/RKF/>.
• 7: Jihie Kim , Jim Blythe, Supporting plan authoring and analysis, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604065]
• 8: Mehrotra, M. "Ontology Analysis for the Semantic Web", AAAI-02 Workshop on Ontologies and the Semantic Web. Edmonton, Canada. July 28-29, 2002.
• 9: Murray, K., 1990. Improving Explanatory Competence, Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, 309-316.
• 10: Kathy Panton , Pierluigi Miraglia , Nancy Salay , Robert C. Kahlert , David Baxter , Roland Reagan, Knowledge formation and dialogue using the KRAKEN toolset, Eighteenth national conference on Artificial intelligence, p.900-905, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 11: Pease, A., and Li, J. "Agent-Mediated Knowledge Engineering Collaboration", in Proceedings of the AAAI 2003 Spring Symposium on Agent-Mediated Knowledge Management. AAAI Technical report SS-03-01, 2003.
• 12: Schrag, R., Pool, M., Chaudhri, V., Kahlert, R., Powers, J., Cohen, P., Fitzgerald, J., and Mishra, S. "Experimental Evaluation of Subject Matter Expert-oriented Knowledge Base Authoring Tools", in NIST Special Publication 990: Performance Metrics for Intelligent Systems, NIST: Proceedings of the 2002 PerMIS Workshop, Gaithersburg, Maryland, pp. 272-279, August, 2002.
• 13: Jérôme Thoméré , Ken Barker , Vinay Chaudhri , Peter Clark , Michael Eriksen , Sunil Mishra , Bruce Porter , Andres Rodriguez, A web-based ontology browsing and editing system, Eighteenth national conference on Artificial intelligence, p.927-934, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 14: Yeh, P., Porter, B., and Barker, K. "Transformation rules for knowledge-based pattern matching", The University of Texas at Austin, Computer Sciences Technical Report UT-AI-TR-03-299. 2003

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.4 Knowledge Representation Formalisms and Methods
          Subjects: Representation languages

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
      I.2.4 Knowledge Representation Formalisms and Methods
          Subjects: Predicate logic

General Terms:
Experimentation, Performance

Keywords:
KRAKEN, NuSketch battlespace, RKF, SHAKEN, evaluation, knowledge acquisition