A design space for multimodal systems: concurrent processing and data fusion

ABSTRACT

Multimodal interaction enables the user to employ different modalities such as voice, gesture and typing for communicating with a computer. This paper presents an analysis of the integration of multiple communication modalities within an interactive system. To do so, a software engineering perspective is adopted. First, the notion of “multimodal system” is clarified. We aim at proving that two main features of a multimodal system are the concurrency of processing and the fusion of input/output data. On the basis of these two features, we then propose a design space and a method for classifying multimodal systems. In the last section, we present a software architecture model of multimodal systems which supports these two salient properties: concurrency of processing and data fusion. Two multimodal systems developed in our team, VoicePaint and NoteBook, are used to illustrate the discussion.

REFERENCES

• 1: Kjelldahl L., Introduction. In Proc. 1st Eurographics Workshop, Stockholm, Sweden (April 18/19,1991), Springer Verlag, pp. 3-5.
• 2: Meera M. Blattner , Roger B. Dannenberg, CHI'90 workshop on multimedia and multimodal interface design, ACM SIGCHI Bulletin, v.22 n.2, p.54-58, Oct. 1990 [doi>10.1145/122475.122486]
• 3: Byte, Special issue on Computing without Keyboard, (July 1990), pp. 202-251.
• 4: Coutaz J. Multimedia and Multimodal User Interfaces: A Taxonomy for Software Engineering Research Issues. in Proc. Second East-Weat HCI conference (St Petersburg, Aug. 1992), pp.229-240.
• 5: Bruce F. Webster, The NeXT book, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 6: Frohlich D.M. The Design Space of Interfaces. Multimedia Systems, Interaction and Applications. In Proc. 1st Eurographics Workshop, Stockholm, Sweden (April 18/19,1991), Springer Verlag, pp. 53-69.
• 7: William Buxton, Lexical and pragmatic considerations of input structures, ACM SIGGRAPH Computer Graphics, v.17 n.1, p.31-37, January 1983 [doi>10.1145/988584.988586]
• 8: Stuart K. Card , Jock D. Mackinlay , George G. Robertson, A morphological analysis of the design space of input devices, ACM Transactions on Information Systems (TOIS), v.9 n.2, p.99-122, April 1991 [doi>10.1145/123078.128726]
• 9: James D. Foley , Victor L. Wallace , Peggy Chan, The human factors of computer graphics interaction techniques, IEEE Computer Graphics and Applications, v.4 n.11, p.13-48, Nov. 1984
• 10: David P. Anderson , Ramesh Govindan , George Homsy, Abstractions for Continuous Media in a Network Window System, University of California at Berkeley, Berkeley, CA, 1990
• 11: Arno P. J. Gourdol , Laurence Nigay , Daniel Salber , Joëlle Coutaz, Two Case Studies of Software Architecture for Multimodal Interactive Systems: VoicePaint and a Voice-enabled Graphical Notebook, Proceedings of the IFIP TC2/WG2.7 Working Conference on Engineering for Human-Computer Interaction, p.271-284, August 10-12, 1992
• 12: Barbara L. Chalfonte , Robert S. Fish , Robert E. Kraut, Expressive richness: a comparison of speech and text as media for revision, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.21-26, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108848]
• 13: Arja Vainio-Larsson , Rebecca Orring, Evaluating the usability of user interfaces: Research in practice, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.323-328, August 27-31, 1990
• 14: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 15: A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop, ACM SIGCHI Bulletin, v.24 n.1, p.32-37, Jan. 1992 [doi>10.1145/142394.142401]
• 16: Nigay L. and Coutaz J. Building User Interfaces: Organizing Software Agents. In Proc. ESPRIT'91 Conference (Bruxelles, Nov. 1991), pp. 707-719.
• 17: J. Bryan Lewis , Lawrence Koved , Daniel T. Ling, Dialogue structures for virtual worlds, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.131-136, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108864]
• 18: James D. Foley , Andries van Dam , Steven K. Feiner , John F. Hughes, Computer graphics: principles and practice (2nd ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1990
• 19: Ralph D. Hill, Supporting concurrency, communication, and synchronization in human-computer interaction—the Sassafras UIMS, ACM Transactions on Graphics (TOG), v.5 n.3, p.179-210, July 1986 [doi>10.1145/24054.24055]
• 20: Wilson M. The first MMI2 Demonstrator, A Multimodal Interface for Man Machine interaction with Knowledge Based Systems. Deliverable D7, ESPRIT project 2474 MMI2, Tech. report Rutherford Appleton Laboratory, Chilton Didcot Oxon OXll 0QX, RAL-91-093, 1991.
• 21: Jean-Michel Lunati , Alexander I. Rudnicky, Spoken language interfaces: the OM system, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.453-454, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108999]
• 22: Coutaz J. PAC: an Implementation Model for Dialog Design, In Proc, Interact'87, (Stuttgart, Sept. 1987), H-J. Bullinger, B. Shackel ed., North Holland, pp. 431-436.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
          Subjects: Natural language interfaces
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis
  I.3 COMPUTER GRAPHICS
      I.3.4 Graphics Utilities
          Subjects: Graphics editors

General Terms:
Algorithms, Design, Human Factors

Keywords:
concurrency, data fusion, design space, modality, multimodal interaction, software architecture, taxonomy