Modality fusion for graphic design applications

ABSTRACT

Users must enter a complex mix of spatial and abstract information when operating a graphic design application. Speech / language provides a fluid and natural method for specifying abstract information while a spatial input device is often most intuitive for the entry of spatial information. Thus, the combined speech / gesture interface is ideally suited to this application domain. While some research has been conducted on multimodal graphic design applications, advanced research on modality fusion has typically focused on map related applications. This paper considers the particular demands of graphic design applications and what impact these demands will have on the general strategies employed when combining the speech and gesture channels. We also describe initial work on our own multimodal graphic design application (DPD) which uses these strategies.

REFERENCES

• 1: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 2: Christopher F. Herot, Graphical input through machine recognition of sketches, Proceedings of the 3rd annual conference on Computer graphics and interactive techniques, p.97-102, July 14-16, 1976, Philadelphia, Pennsylvania
• 3: Takeo Igarashi , John F. Hughes, A suggestive interface for 3D drawing, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502379]
• 4: Takeo Igarashi , John F. Hughes, Clothing manipulation, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.571999]
• 5: Takeo Igarashi , Satoshi Matsuoka , Hidehiko Tanaka, Teddy: a sketching interface for 3D freeform design, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.409-416, July 1999 [doi>10.1145/311535.311602]
• 6: Takeo Igarashi , Satoshi Matsuoka , Sachiko Kawachiya , Hidehiko Tanaka, Interactive beautification: a technique for rapid geometric design, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.105-114, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263525]
• 7: Michael Johnston, Deixis and conjunction in multimodal systems, Proceedings of the 18th conference on Computational linguistics, p.362-368, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990873]
• 8: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 9: L. Julia , C. Faure, Pattern recognition and beautification for a pen based interface, Proceedings of the Third International Conference on Document Analysis and Recognition (Volume 1), p.58, August 14-15, 1995
• 10: Kaiser, C., and Cohen, P.R. Implementation testing of a hybrid symbolic/statistical multimodal architecture, In Proceedings of the International Conference on spoken language processing (Denver, September 2002,) pp. 173--176.
• 11: James A. Landay , Brad A. Myers, Interactive sketching for the early stages of user interface design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.43-50, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223910]
• 12: Laurence Nigay , Joëlle Coutaz, A design space for multimodal systems: concurrent processing and data fusion, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.172-178, May 1993, Amsterdam, The Netherlands
• 13: Nishimoto, T., Shida, N., Kobayashi T., and Shirai, K. Multimodal Drawing Tool Using Speech, Mouse and Key-Board. In Proceedings 1994 International Conference on Spoken Language Processing (ICSLP), vol. 3:1287--1290.
• 14: Makoto Okabe , Takeo Igarashi, 3D modeling of trees from freehand sketches, Proceedings of the SIGGRAPH 2003 conference on Sketches & applications: in conjunction with the 30th annual conference on Computer graphics and interactive techniques, July 27-31, 2003, San Diego, California [doi>10.1145/965400.965565]
• 15: Sharon Oviatt, Multimodal system processing in mobile environments, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.21-30, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354408]
• 16: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 17: Pausch, R. and Leatherby, J.H. An Empirical Study: Adding Voice Input to a Graphical Editor. Journal of the American Voice Input/Output Society, 9:2, July, 1991, pp 55--66.
• 18: Pitel, G., Sansonnet, J. A differential representation of predicates for extensional reference resolution. In proceedings of ARQAS, (Venice 2003).
• 19: Rowe J. Speech recognition takes us a giant step closer to more natural interaction with CAD software Computer Graphics World February, 2001
• 20: Sato, T., and Tojo, A. recognition and understanding of hand-drawn diagrams. In Proceedings of 6th International Conference on pattern recognition. IEEE Computer Society Press, New Jersey, 1982.
• 21: Eric Saund , Thomas P. Moran, A perceptually-supported sketch editor, Proceedings of the 7th annual ACM symposium on User interface software and technology, p.175-184, November 02-04, 1994, Marina del Rey, California, United States [doi>10.1145/192426.192494]
• 22: Damien Touraine , Patrick Bourdot , Yacine Bellik , Laurence Bolot, A framework to manage multimodal fusion of events for advanced interactions within virtual environments, Proceedings of the workshop on Virtual environments 2002, May 30-31, 2002, Barcelona, Spain
• 23: Vo, T., and Wood, C. Building an application framework for speech and pen input integration in multimodal learning interfaces. In Proceedings of the International Conference on acoustics speech and signal processing (IEEE-ICAASP 1996), Vol. 6, 3545--3548, IEEE Press.
• 24: Wolfgang Wahlster, User and discourse models for multimodal communication, Intelligent user interfaces, ACM Press, New York, NY, 1991 [doi>10.1145/107215.128691]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Natural language; Graphical user interfaces (GUI)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding

General Terms:
Algorithms

Keywords:
graphic design, modality fusion, multimodal interface, pen interface, speech input