A framework for rapid development of multimodal interfaces

ABSTRACT

Despite the availability of multimodal devices, there are very few commercial multimodal applications available. One reason for this may be the lack of a framework to support development of multimodal applications in reasonable time and with limited resources. This paper describes a multimodal framework enabling rapid development of applications using a variety of modalities and methods for ambiguity resolution, featuring a novel approach to multimodal fusion. An example application is studied that was created using the framework.

REFERENCES

• 1: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, Proceedings of the 7th annual conference on Computer graphics and interactive techniques, p.262-270, July 14-18, 1980, Seattle, Washington, United States
• 2: L. Boves and E. den Os. Multimodal multilingual information services for small mobile terminals (MUST). Technical report, Eurescom, 2002. http://www.eurescom.de/pub/deliverables/documents/P1100-series/P1104/p1104-d1.pdf.
• 3: CAIP. DISCIPLE-mobile computing and collaboration. http://www.caip.rutgers.edu/disciple.
• 4: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 5: Mohamed Fayad , Douglas C. Schmidt, Object-oriented application frameworks, Communications of the ACM, v.40 n.10, p.32-38, Oct. 1997 [doi>10.1145/262793.262798]
• 6: A Real-Time Framework for Natural Multimodal Interaction with Large Screen Displays, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.349, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167020]
• 7: J. A. Larson and T. V. Raman. W3C multimodal interaction framework. http://www.w3.org/TR/mmi-framework, 2 December 2002. W3C Note.
• 8: Marcus Eduardo Markiewicz , Carlos J. P. de Lucena, Object oriented framework development, Crossroads, v.7 n.4, p.3-9, Summer 2001 [doi>10.1145/372765.372771]
• 9: B. Shneiderman. The future of interactive systems and the emergence of direct manipulation. Behaviour and Information Technology, 1(3):237--256, 1982.
• 10: The Apache Jakarta Project. Jakarta BSF -- bean scripting framework. http://jakarta.apache.org/bsf.
• 11: D. Toledano, S. Wang, S. Cyphers, and J. Glass. Extending the galaxy communicator architecture for multimodal interaction research. submitted to ACM Trans. on Human-Computer Interaction, Aug 2002.
• 12: John M. Vlissides , Mark A. Linton, Unidraw: a framework for building domain-specific graphical editors, ACM Transactions on Information Systems (TOIS), v.8 n.3, p.237-268, July 1990 [doi>10.1145/98188.98197]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language; Voice I/O

General Terms:
Algorithms, Design, Human Factors

Keywords:
application frameworks, command and control, direct manipulation, multimodal fusion, multimodal interfaces