Sketching for military courses of action diagrams

ABSTRACT

A serious barrier to the digitalization of the US military is that commanders find traditional mouse/menu, CAD-style interfaces unnatural. Military commanders develop and communicate battle plans by sketching courses of action (COAs). This paper describes nuSketch Battlespace, the latest version in an evolving line of sketching interfaces that commanders find natural, yet supports significant increased automation. We describe techniques that should be applicable to any specialized sketching domain: glyph bars and compositional symbols to tractably handle the large number of entities that military domains use, specialized glyph types and gestures to keep drawing tractable and natural, qualitative spatial reasoning to provide sketch-based visual reasoning, and comic graphs to describe multiple states and plans. Experiments, both completed and in progress, are described to provide evidence as to the utility of the system.

REFERENCES

• 1: Alvarado, Christine and Davis, Randall (2001). Resolving ambiguities to create a natural sketch based interface. Proceedings of IJCAI-2001, August 2001.
• 2: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 3: Anthony G. Cohn, Calculi for Qualitative Spatial Reasoning, Proceedings of the International Conference AISMC-3 on Artificial Intelligence and Symbolic Mathematical Computation, p.124-143, September 23-25, 1996
• 4: Kenny R. Coventry, Spatial prepositions, functional relations, and lexical specification, Representation and processing of spatial expressions, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1998
• 5: Davis, I. 2000. Warp Speed: Path Planning for Star Trek: Armada, AI and Interactive Entertainment: Papers from the 2002 AAAI Spring Symp., AAAI Press, Menlo Park, Calif.
• 6: Geoffrey Edwards , Bernard Moulin, Toward the simulation of spatial mental images using the Voronoi¨ model, Representation and processing of spatial expressions, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1998
• 7: Brian Falkenhainer , Kenneth D. Forbus , Dedre Gentner, The structure-mapping engine: algorithm and examples, Artificial Intelligence, v.41 n.1, p.1-63, Nov. 1989 [doi>10.1016/0004-3702(89)90077-5]
• 8: Feist, M. and Gentner, D. 1998. On Plates, Bowls and Dishes: Factors in the Use of English IN and ON. Proceedings of the 20th annual meeting of the Cognitive Science Society
• 9: Ferguson, R.W., Rasch, R.A., Turmel, W., & Forbus, K.D. (2000) Qualitative Spatial Interpretation of Course-of-Action Diagrams. Proceedings of the 14th International Workshop on Qualitative Reasoning. Morelia, Mexico. June, 2000.
• 10: Forbus, K. 1989. Introducing actions into qualitative simulation", Proceedings of IJCAI-89, August.
• 11: Forbus, K. 1995. Qualitative Spatial Reasoning: Framework and Frontiers. In Glasgow, J., Narayanan, N., and Chandrasekaran, B. Diagrammatic Reasoning: Cognitive and Computational Perspectives. MIT Press, pp. 183--202.
• 12: Kenneth D. Forbus , Johan de Kleer, Building problem solvers, MIT Press, Cambridge, MA, 1993
• 13: Forbus, K., Ferguson, R. and Gentner, D. (1994) Incremental structure-mapping. Proceedings of the Cognitive Science Society, August.
• 14: Kenneth D. Forbus , Ronald W. Ferguson , Jeffery M. Usher, Towards a computational model of sketching, Proceedings of the 6th international conference on Intelligent user interfaces, p.77-83, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360278]
• 15: Forbus, K., Gentner, D. and Law, K. 1995. MAC/FAC: A model of Similarity-based Retrieval. Cognitive Science, 19(2), April-June, pp 141--205.
• 16: Forbus, K., Mahoney, J.V., and Dill, K. 2001. How qualitative spatial reasoning can improve strategy game AIs: A preliminary report. 15th International workshop on Qualitative Reasoning (QR01), San Antonio, Texas, May.
• 17: Kenneth D. Forbus , Paul Nielsen , Boi Faltings, Qualitative spatial reasoning: the CLOCK project, Artificial Intelligence, v.51 n.1-3, p.417-471, Oct. 1991 [doi>10.1016/0004-3702(91)90116-2]
• 18: Kenneth D. Forbus , Jeffrey Usher, Sketching for knowledge capture: a progress report, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502730]
• 19: Gross, M. (1996) The Electronic Cocktail Napkin - computer support for working with diagrams. Design Studies. 17(1), 53--70.
• 20: Gross, M. and Do, E. (1995) Drawing Analogies - Supporting Creative Architectural Design with Visual References. in 3d International Conference on Computational Models of Creative Design, M-L Maher and J. Gero (eds), Sydney: University of Sydney, 37--58.
• 21: King, Gary W., Brent Heeringa, Joe Catalano, David L. Westbrook, and Paul Cohen. Models of Defeat. Proceedings of the Second International Conference on Knowledge Systems for Colation Operations 2002. pp. 85--90.
• 22: James A. Landay , Brad A. Myers, Sketching storyboards to illustrate interface behaviors, Conference companion on Human factors in computing systems: common ground, p.193-194, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257257]
• 23: David R. McGee , Philip R. Cohen, Creating tangible interfaces by augmenting physical objects with multimodal language, Proceedings of the 6th international conference on Intelligent user interfaces, p.113-119, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360305]
• 24: David R. McGee , Philip R. Cohen , R. Matthews Wesson , Sheilah Horman, Comparing paper and tangible, multimodal tools, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503449]
• 25: Robert Rasch , Alexander Kott , Kenneth D. Forbus, AI on the battlefield: an experimental exploration, Eighteenth national conference on Artificial intelligence, p.906-912, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 26: Stahovich, T. F., Davis, R., and Shrobe, H., "Generating Multiple New Designs from a Sketch," in Proceedings Thirteenth National Conference on Artificial Intelligence, AAAI-96, pp. 1022--29, 1996.
• 27: A. Waibel , B. Suhm , . M. Vo , J. Yang, Multimodal Interfaces for Multimedia Information Agents, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97) -Volume 1, p.167, April 21-24, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Perceptual reasoning
      I.2.4 Knowledge Representation Formalisms and Methods

General Terms:
Algorithms, Design, Human Factors

Keywords:
analogy, multimodal interfaces, nuSketch, qualitative reasoning, sketch understanding, spatial reasoning