ICARE: a component-based approach for the design and development of multimodal interfaces

ABSTRACT

Multimodal interactive systems support multiple interaction techniques such as the synergistic use of speech, gesture and eye gaze tracking. The flexibility they offer results in an increased complexity that current software development tools do not address appropriately. In this paper we describe a component-based approach, called ICARE, for specifying and developing multimodal interfaces. Our approach relies on two types of components: (i) elementary components that describe pure modalities and (ii) composition components (Complementarity, Redundancy and Equivalence) that enable the designer to specify combined usage of modalities. The designer graphically assembles the ICARE components and the code of the multimodal user interface is automatically generated. Although the ICARE platform is not fully developed, we illustrate the applicability of the approach with the implementation of two multimodal systems: MEMO a GeoNote system and MID, a multimodal identification interface.

REFERENCES

• 1: Bass, L. et al. Market Assessment of Component-Based Software Engineering. SEI TR (2000).
• 2: Bernsen, N. Modality Theory in support of multimodal interface design. Proc. of Intelligent Multi-Media Multi-Modal Systems (1994), 37--44.
• 3: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 4: Dubois, E., Nigay, L., Troccaz, J. Assessing Continuity and Compatibility in Augmented Reality Systems. UAIS Journal, 4 (2002), 263--273.
• 5: Beverly L. Harrison , Kenneth P. Fishkin , Anuj Gujar , Carlos Mochon , Roy Want, Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.17-24, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274647]
• 6: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 7: JavaBeans1.01 specification, Sun Microsystems (1997), java.sun.com/products/javabeans/docs/
• 8: Martin, J. C. TYCOON: Theoretical Framework and Software Tools for Multimodal Interfaces. Intelligence and Multimodality in Multimedia Interfaces, AAAI Press (1997).
• 9: Laurence Nigay , Joëlle Coutaz, A generic platform for addressing the multimodal challenge, Proceedings of the SIGCHI conference on Human factors in computing systems, p.98-105, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223917]
• 10: Nigay, L., Coutaz, J. Multifeature Systems: The CARE Properties and Their Impact on Software Design. Intelligence and Multimodality in Multimedia Interfaces, AAAI Press (1997).
• 11: Sharon Oviatt , Philip Cohen, Perceptual user interfaces: multimodal interfaces that process what comes naturally, Communications of the ACM, v.43 n.3, p.45-53, March 2000 [doi>10.1145/330534.330538]
• 12: Sharon Oviatt, Taming recognition errors with a multimodal interface, Communications of the ACM, v.43 n.9, p.45-51, Sept. 2000 [doi>10.1145/348941.348979]
• 13: Oviatt, S. et al. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. HCI, 15, 4 (2000), 263--322.
• 14: Per Persson , Fredrik Espinoza , Elenor Cacciatore, GeoNotes: social enhancement of physical space, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634097]
• 15: A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop, ACM SIGCHI Bulletin, v.24 n.1, p.32-37, Jan. 1992 [doi>10.1145/142394.142401]
• 16: Matthew Turk , George Robertson, Perceptual user interfaces (introduction), Communications of the ACM, v.43 n.3, p.32-34, March 2000 [doi>10.1145/330534.330535]
• 17: Tracy Westeyn , Helene Brashear , Amin Atrash , Thad Starner, Georgia tech gesture toolkit: supporting experiments in gesture recognition, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958452]
• 18: Gilles Privat , Claude Kintzig , Gerard Poulain, Communicating with Smart Objects: Developing Technology for Usable Persuasive Computing Systems, Kogan Page Science, 2003

INDEX TERMS

Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping

General Terms:
Algorithms, Human Factors

Keywords:
multimodal interactive systems, software components