Naturally conveyed explanations of device behavior

ABSTRACT

Designers routinely explain their designs to one another using sketches and verbal descriptions of behavior, both of which can be understood long before the device has been fully specified. But current design tools fail almost completely to support this sort of interaction, instead not only forcing designers to specify details of the design, but typically requiring that they do so by navigating a forest of menus and dialog boxes, rather than directly describing the behaviors with sketches and verbal explanations. We have created a prototype system, called assistance, capable of interpreting multimodal explanations for simple 2-D kinematic devices. The program generates a model of the events and the causal relationships between events that have been described via hand drawn sketches, sketched annotations, and verbal descriptions. Our goal is to make the designer's interaction with the computer more like interacting with another designer. This requires the ability not only to understand physical devices but also to understand the means by which the explanations of these devices are conveyed.

REFERENCES

• 1: C. Alvarado and R. Davis. Resolving ambiguities to create a natural computer-based sketching environment. In IJCAI-01, 2001.
• 2: V. Baya and L. Leifer. Understanding information management in conceptual design. In Analysing Design Activity, pages 151--168. John Wiley, 1996.
• 3: M. Bett, R. Gross, H. Yu, X. Zhu, Y. Pan, J. Yang, and A. Waibel. Multimodal meeting tracker. In Proceedings of RIAO2000, April 2000.
• 4: Gary C. Borchardt, Causal Reconstruction, Massachusetts Institute of Technology, Cambridge, MA, 1993
• 5: Matthew Brand, Physics-based visual understanding, Computer Vision and Image Understanding, v.65 n.2, p.192-205, Feb. 1997 [doi>10.1006/cviu.1996.0572]
• 6: Tzachi Dar , Leo Joskowicz , Ehud Rivlin, Understanding mechanical motion: from images to behaviors, Artificial Intelligence, v.112 n.1-2, p.147-179, Aug. 1999 [doi>10.1016/S0004-3702(99)00040-5]
• 7: Kenneth D. Forbus , Johan de Kleer, Building problem solvers, MIT Press, Cambridge, MA, 1993
• 8: Kenneth D. Forbus , Ronald W. Ferguson , Jeffery M. Usher, Towards a computational model of sketching, Proceedings of the 6th international conference on Intelligent user interfaces, p.77-83, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.360278]
• 9: Marti A. Hearst, Sketching Intelligent Systems, IEEE Intelligent Systems, v.13 n.3, p.10-19, May 1998 [doi>10.1109/5254.683174]
• 10: B. Katz. From sentence processing to information access on the world wide web. In AAAI Spring Symposium on Natural Language Processing for the World Wide Web, 1997.
• 11: David R. McGee , Philip R. Cohen , Lizhong Wu, Something from nothing: augmenting a paper-based work practice via multimodal interaction, Proceedings of DARE 2000 on Designing augmented reality environments, p.71-80, April 2000, Elsinore, Denmark [doi>10.1145/354666.354674]
• 12: N. Narayanan, M. Suwa, and H. Motoda. Hypothesizing Behavior from Device Diagrams, chapter 15, pages 501--534. MIT Press, Cambridge, MA, 1995.
• 13: S. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. In Human Computer Interaction, volume 15, pages 263--322. Addison-Wesley, 2000.
• 14: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 15: Martha S. Palmer , Rebecca J. Passonneau , Carl Weir , Tim Finin, The KERNEL text understanding system, Artificial Intelligence, v.63 n.1-2, p.17-68, Oct. 1993 [doi>10.1016/0004-3702(93)90014-3]
• 16: T. Stahovich, R. Davis, and H. Shrobe. Generating multiple new designs from a sketch. In Proc. 13th AAAI, pages 1022--1030, Menlo Park, Aug. 1996. MIT Press.
• 17: D. Ullman, S. Wood, and D. Craig. The importance of drawing in the mechanical design process. Computers and Graphics, 14(2):263--274, 1990.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.6 COMPUTER-AIDED ENGINEERING
      Subjects: Computer-aided design (CAD)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Human Factors