The WAMI toolkit for developing, deploying, and evaluating web-accessible multimodal interfaces

ABSTRACT

Many compelling multimodal prototypes have been developed which pair spoken input and output with a graphical user interface, yet it has often proved difficult to make them available to a large audience. This unfortunate reality limits the degree to which authentic user interactions with such systems can be collected and subsequently analyzed. We present the WAMI toolkit, which alleviates this difficulty by providing a framework for developing, deploying, and evaluating Web-Accessible Multimodal Interfaces in which users interact using speech, mouse, pen, and/or touch. The toolkit makes use of modern web-programming techniques, enabling the development of browser-based applications which rival the quality of traditional native interfaces, yet are available on a wide array of Internet-connected devices. We will showcase several sophisticated multimodal applications developed and deployed using the toolkit, which are available via desktop, laptop, and tablet PCs, as well as via several mobile devices. In addition, we will discuss resources provided by the toolkit for collecting, transcribing, and annotating usage data from multimodal user interactions.

REFERENCES

• 1: A. Acero, N. Bernstein, R. Chambers, Y. C. Jui, X. Li, J. Odell, P. Nguyen, O. Scholz, and G. Zweig. Live search for mobile: Web services by voice on the cellphone. In Proc. of ICASSP, 2008.
• 2: G. Aist, J. Allen, E. Campana, C. G. Gallo, S. Stoness, M. Swift, and M. K. Tanenhaus. Incremental understanding in human-computer dialogue and experimental evidence for advantages over nonincremental methods. In R. Artstein and L. Vieu, editors, Proc. of the 11th Workshop on the Semantics and Pragmatics of Dialogue, pages 149--154, 2007.
• 3: J. Axelsson, C. Cross, J. Ferrans, G. McCobb, T. V. Raman, and L. Wilson. Mobile X V 1.2. Technical report, 2005. http://www.voicexml.org/specs/multimodal/xv/mobile/12/.
• 4: A. Gruenstein, B.-J. P. Hsu, J. Glass, S. Seneff, L. Hetherington, S. Cyphers, I. Badr, C. Wang, and S. Liu. A multimodal home entertainment interface via a mobile device. In Proc. of the ACL Workshop on Mobile Language Processing, 2008.
• 5: A. Gruenstein, S. Seneff, and C. Wang. Scalable and portable web-based multimodal dialogue interaction with geographical databases. In Proc. of INTERSPEECH, 2006.
• 6: A. Hjalmarsson. Evaluating AdApt, a multi-modal conversational dialogue system using PARADISE. Master's thesis, KTH, Stockhom, Sweden, 2002.
• 7: Mikko Honkala , Mikko Pohja, Multimodal interaction with xforms, Proceedings of the 6th international conference on Web engineering, July 11-14, 2006, Palo Alto, California, USA [doi>10.1145/1145581.1145624]
• 8: Java speech grammar format. http://java.sun.com/products/java-media/speech/forDevelopers/JSGF/.
• 9: Michael Johnston , Srinivas Bangalore , Gunaranjan Vasireddy , Amanda Stent , Patrick Ehlen , Marilyn Walker , Steve Whittaker , Preetam Maloor, MATCH: an architecture for multimodal dialogue systems, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073146]
• 10: Kouichi Katsurada , Yusaku Nakamura , Hirobumi Yamada , Tsuneo Nitta, XISL: a language for describing multimodal interaction scenarios, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958483]
• 11: Raymond Lau , Giovanni Flammia , Christine Pao , Victor Zue, WebGALAXY: beyond point and click—a conversational interface to a browser, Computer Networks and ISDN Systems, v.29 n.8-13, p.1385-1393, Sep. 1997
• 12: Oliver Lemon , Alexander Gruenstein, multithreaded context for robust conversational interfaces: Context-sensitive speech recognition and interpretation of corrective fragments, ACM Transactions on Computer-Human Interaction (TOCHI), v.11 n.3, p.241-267, September 2004 [doi>10.1145/1017494.1017496]
• 13: I. McGraw and S. Seneff. Speech-enabled card games for language learners. In Proc. of the 23rd AAAI Conference on Artificial Intelligence, 2008.
• 14: A. Raux, D. Bohus, B. Langner, A. Black, and M. Eskenazi. Doing research on a deployed spoken dialogue system: One year of Let's Go! experience. In Proc. of INTERSPEECH-ICSLP, 2006.
• 15: S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue. Galaxy-II: A reference architecture for conversational system development. In Proc. ICSLP, 1998.
• 16: Amanda Stent , John Dowding , Jean Mark Gawron , Elizabeth Owen Bratt , Robert Moore, The CommandTalk spoken dialogue system, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.183-190, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034713]
• 17: Tellme mobile. http://m.tellme.com.
• 18: K. Wang. SALT: A spoken language interface for web-based multimodal dialog systems. In Proc. of the 7th International Conference on Spoken Language Processing, 2002.
• 19: Yahoo onesearch. http://mobile.yahoo.com.
• 20: V. Zue, S. Seneff, J. Glass, J. Polifroni, C. Pao, T. J. Hazen, and L. Hetherington. JUPITER: A telephone-based conversational interface for weather information. IEEE Transactions on Speech and Audio Processing, 8(1), January 2000.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Voice I/O; Natural language

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis; Language parsing and understanding

General Terms:
Design, Experimentation

Keywords:
dialogue system, multimodal interface, speech recognition, voice over ip, world wide web