Sketching for knowledge capture: a progress report

ABSTRACT

Many concepts and situations are best explained by sketching. This paper describes our work on sKEA, the sketching Knowledge Entry Associate, a system designed for knowledge capture via sketching. We discuss the key ideas of sKEA: blob semantics for glyphs to sidestep recognition for visual symbols, qualitative spatial reasoning to provide richer visual and conceptual understanding of what is being communicated, arrows to express domain relationships, layers to express within-sketch segmentation (including a meta-layer to express subsketch relationships themselves via sketching), and analogical comparison to explore similarities and differences between sketched concepts. Experiences with sKEA to date and future plans are also discussed.

REFERENCES

• 1: Allen, J.F. et al, The TRAINS Project: A Case Study in Defining a Conversational Planning Agent, Journal of Experimental and Theoretical AI, 1995.
• 2: Alvarado, Christine and Davis, Randall (2001). Resolving ambiguities to create a natural sketch based interface. Proceedings of IJCAI-2001, August 2001.
• 3: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 4: Clark, H. H. 1996. Using language. Cambridge University Press.
• 5: Clark, H.H. 1999. Speaking in Time. Proceedings of the ESCA workshop on dialogue and prosody, September 1- 3, Veldhoven, the Netherlands.
• 6: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 7: Anthony G. Cohn, Calculi for Qualitative Spatial Reasoning, Proceedings of the International Conference AISMC-3 on Artificial Intelligence and Symbolic Mathematical Computation, p.124-143, September 23-25, 1996
• 8: Kenny R. Coventry, Spatial prepositions, functional relations, and lexical specification, Representation and processing of spatial expressions, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1998
• 9: Geoffrey Edwards , Bernard Moulin, Toward the simulation of spatial mental images using the Voronoi¨ model, Representation and processing of spatial expressions, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1998
• 10: Egenhofer, M. (1997) Query Processing in Spatial-Queryby- Sketch in Journal of Visual Languages and Computing 8(4), 403-424 pp.
• 11: Brian Falkenhainer , Kenneth D. Forbus , Dedre Gentner, The structure-mapping engine: algorithm and examples, Artificial Intelligence, v.41 n.1, p.1-63, Nov. 1989 [doi>10.1016/0004-3702(89)90077-5]
• 12: Feist, M. and Gentner, D. 1998. On Plates, Bowls and Dishes: Factors in the Use of English IN and ON. Proceedings of the 20th annual meeting of the Cognitive Science Society
• 13: Ronald W. Ferguson , Kenneth D. Forbus, GeoRep: A Flexible Tool for Spatial Representation of Line Drawings, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.510-516, July 30-August 03, 2000
• 14: Ferguson, R.W., Rasch, R.A., Turmel, W., & Forbus, K.D. (2000) Qualitative Spatial Interpretation of Courseof- Action Diagrams. Proceedings of the 14th International Workshop on Qualitative Reasoning. Morelia, Mexico. June, 2000.
• 15: Forbus, K. 1995. Qualitative Spatial Reasoning: Framework and Frontiers. In Glasgow, J., Narayanan, N., and Chandrasekaran, B. Diagrammatic Reasoning: Cognitive and Computational Perspectives. MIT Press, pp. 183-202.
• 16: Forbus, K. 2000. Exploring analogy in the large. In Gentner, D., Holyoak, K. and Kokinov, B. (Eds) Analogy: Perspectives from Cognitive Science. Cambridge, MA: MIT Press.
• 17: Kenneth D. Forbus , Johan de Kleer, Building problem solvers, MIT Press, Cambridge, MA, 1993
• 18: Forbus, K., Ferguson, R. and Gentner, D. (1994) Incremental structure-mapping. Proceedings of the Cognitive Science Society, August.
• 19: Kenneth D. Forbus , Ronald W. Ferguson , Jeffery M. Usher, Towards a computational model of sketching, Proceedings of the 6th international conference on Intelligent user interfaces, p.77-83, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360278]
• 20: Forbus, K., Mahoney, J.V., and Dill, K. 2001. How qualitative spatial reasoning can improve strategy game AIs: A preliminary report. 15th International workshop on Qualitative Reasoning (QR01), San Antonio, Texas, May.
• 21: Kenneth D. Forbus , Paul Nielsen , Boi Faltings, Qualitative spatial reasoning: the CLOCK project, Artificial Intelligence, v.51 n.1-3, p.417-471, Oct. 1991 [doi>10.1016/0004-3702(91)90116-2]
• 22: Gentner, D. and Markman, A. 1997. Structure Mapping in Analogy and Similarity. American Psychologist, January, pp 45-56
• 23: Gross, M. (1996) The Electronic Cocktail Napkin - computer support for working with diagrams. Design Studies. 17(1), 53-70.
• 24: Gross, M. and Do, E. (1995) Drawing Analogies - Supporting Creative Architectural Design with Visual References. in 3d International Conference on Computational Models of Creative Design, M-L Maher and J. Gero (eds), Sydney: University of Sydney, 37-58.
• 25: James A. Landay , Brad A. Myers, Sketching storyboards to illustrate interface behaviors, Conference companion on Human factors in computing systems: common ground, p.193-194, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257257]
• 26: Mark T. Maybury , Wolfgang Wahlster, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 27: J.D. Novak and D.B. Gowin, Learning How To Learn, New York: Cambridge University Press. 1984.
• 28: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 29: E. Saund, Symbolic Construction of a 2-D Scale-Space Image, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.12 n.8, p.817-830, August 1990 [doi>10.1109/34.57672]
• 30: E. Saund , T. P. Moran, Perceptual organization in an interactive sketch editing application, Proceedings of the Fifth International Conference on Computer Vision, p.597, June 20-23, 1995
• 31: Stahovich, T. F., Davis, R., and Shrobe, H., "Generating Multiple New Designs from a Sketch," in Proceedings Thirteenth National Conference on Artificial Intelligence, AAAI-96, pp. 1022-29, 1996.
• 32: A. Waibel , B. Suhm , . M. Vo , J. Yang, Multimodal Interfaces for Multimedia Information Agents, Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97) -Volume 1, p.167, April 21-24, 1997

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Modeling and recovery of physical attributes

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.1 Project and People Management
          Subjects: Systems analysis and design

General Terms:
Algorithms, Design, Experimentation, Performance, Theory

Keywords:
analogy, artificial intelligence sketching, diagrammatic reasoning, knowledge acquisition, qualitative modeling, sketch understanding, spatial reasoning