Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

This paper presents a formal framework for multimodal dialogue systems by applying a set of complexity reduction patterns. The minimalist approach described in this paper combines recursive application of Model-View-Controller (MVC) design patterns with layering and interpretation. It leads to a modular, concise, flexible and dynamic framework building upon a few core constructs. This framework could expedite the development of complex multimodal dialogue systems with sound software development practices and techniques. A XML based prototype multimodal dialogue system that embodies this framework is developed and studied. Experimental results indicate that the proposed framework is effective and well suited for multimodal interaction in complex business transactions.

REFERENCES

• 1: Stock, O., Zancanaro, M. (eds.): Multimodal Intelligent Information Presentation. Series Text, Speech and Language Technology, Vol 27. Springer, 2005.
• 2: Adrian Stanciulescu , Quentin Limbourg , Jean Vanderdonckt , Benjamin Michotte , Francisco Montero, A transformational approach for multimodal web user interfaces based on UsiXML, Proceedings of the 7th international conference on Multimodal interfaces, October 04-06, 2005, Torento, Italy [doi>10.1145/1088463.1088508]
• 3: Gregor Möhler, Modeling Output in the EMBASSI Multimodal Dialog System, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.111, October 14-16, 2002 [doi>10.1109/ICMI.2002.1166978]
• 4: Daniel Jurafsky , James H. Martin, Speech and Language Processing (2nd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2006
• 5: Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, 1995. Design Patterns. Addison Wesley, 1995.
• 6: EMMA: Extensible MultiModal Annotation markup language, W3C Candidate Recommendation, 11 December 2007.
• 7: Jullien Bouchet , Laurence Nigay, ICARE: a component-based approach for the design and development of multimodal interfaces, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986055]
• 8: Li Li, Quanzhi Li, Wu Chou, Feng Liu, 2007. R-Flow: An Extensible XML Based Multimodal Dialogue System Architecture, Proceedings of 9th IEEE International Workshop on Multimedia Signal Processing (MMSP 2007), page 86--89, 2007.
• 9: Mikko Honkala , Mikko Pohja, Multimodal interaction with xforms, Proceedings of the 6th international conference on Web engineering, July 11-14, 2006, Palo Alto, California, USA [doi>10.1145/1145581.1145624]
• 10: MMAI: Multimodal Architecture and Interfaces, W3C Working Draft, 14 April 2008.
• 11: SALT: SALTForum, http://www.saltforum.org/
• 12: SCXML: State Chart XML (SCXML): State Machine Notation for Control Abstraction 1.0, W3C Working Draft, 16 May 2008.
• 13: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 14: SMIL: Synchronized Multimedia Integration Language (SMIL 3.0), W3C Candidate Recommendation, 15 January 2008.
• 15: Stephane H. Maes, Chummun Ferial, 2001. Multi-Modal Browser Architecture, presentation (T2-010705) on 3GPP T2 meeting, September 2001.
• 16: VXML v2.1: Voice Extensible Markup Language (VoiceXML) 2.1, W3C Recommendation, 19 June 2007
• 17: VXML v3.0: Sneak Preview: VoiceXML 3.0, www.w3.org/Voice/2006/voicexml3.pdf
• 18: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 19: Gerd Herzog , Heinz Kirchmann , Stefan Merten , Alassane Ndiaye , Peter Poller, MULTIPLATFORM testbed: an integration platform for multimodal dialog systems, Proceedings of the HLT-NAACL 2003 workshop on Software engineering and architecture of language technology systems, p.75-82, May 31-31, 2003 [doi>10.3115/1119226.1119237]
• 20: Bunt, Harry, et al. Fusion and Coordination for Multimodal Interactive Information Presentation. In {1}, 2005
• 21: Ed Kaiser , Alex Olwal , David McGee , Hrvoje Benko , Andrea Corradini , Xiaoguang Li , Phil Cohen , Steven Feiner, Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958438]
• 22: Dumas B. et al. Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces. TEI 2008, February 18--20, 2008.
• 23: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 24: Larson, James. et al. (eds). W3C Multimodal Interaction Framework, W3C NOTE, 06 May 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Prototyping

General Terms:
Design, Languages, Theory

Keywords:
dialogue, multimodal, mvc, xml