MATCH: an architecture for multimodal dialogue systems

ABSTRACT

Mobile interfaces need to allow the user and system to adapt their choice of communication modes according to user preferences, the task at hand, and the physical and social environment. We describe a multimodal application architecture which combines finite-state multimodal language processing, a speech-act based multimodal dialogue manager, dynamic multimodal output generation, and user-tailored text planning to enable rapid prototyping of multimodal interfaces with flexible input and adaptive output. Our testbed application MATCH (Multimodal Access To City Help) provides a mobile multimodal speech-pen interface to restaurant and sub-way information for New York City.

REFERENCES

• 1: J. Alexandersson and T. Becker. 2001. Overlay as the basic operation for discourse processing in a multimodal dialogue system. In 2nd IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems.
• 2: James Allen , Donna Byron , Myroslava Dzikovska , George Ferguson , Lucian Galescu , Amanda Stent, An architecture for a generic dialogue shell, Natural Language Engineering, v.6 n.3-4, p.213-228, September 2000 [doi>10.1017/S135132490000245X]
• 3: E. André. 2002. Natural language in multimedia/multimodal systems. In Ruslan Mitkov, editor, Handbook of Computational Linguistics. OUP.
• 4: G. Carenini and J. D. Moore. 2001. An empirical study of the influence of user tailoring on evaluative argument effectiveness. In IJCAI, pages 1307--1314.
• 5: Michael Johnston , Srinivas Bangalore, Finite-state multimodal parsing and understanding, Proceedings of the 18th conference on Computational linguistics, p.369-375, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990874]
• 6: M. Johnston and S. Bangalore. 2001. Finite-state methods for multimodal parsing and integration. In ESSLLI Workshop on Finite-state Methods, Helsinki, Finland.
• 7: Michael Johnston, Deixis and conjunction in multimodal systems, Proceedings of the 18th conference on Computational linguistics, p.362-368, July 31-August 04, 2000, Saarbrücken, Germany [doi>10.3115/990820.990873]
• 8: S. Larsson, P. Bohlin, J. Bos, and D. Traum. 1999. TrindiKit manual. Technical report, TRINDI Deliverable D2.2.
• 9: D. Martin, A. Cheyer, and D. Moran. 1999. The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence, 13(1--2):91--128.
• 10: M-J. Nederhof. 1997. Regular approximations of CFLs: A grammatical view. In Proceedings of the International Workshop on Parsing Technology, Boston.
• 11: S. L. Oviatt and J. Clow. 1998. An automated tool for analysis of multimodal system performance. In Proceedings of ICSLP.
• 12: Charles Rich , Candace L. Sidner, COLLAGEN: A Collaboration Manager for Software Interface Agents, User Modeling and User-Adapted Interaction, v.8 n.3-4, p.315-350, 1998 [doi>10.1023/A:1008204020038]
• 13: Dean Rubine, Specifying gestures by example, ACM SIGGRAPH Computer Graphics, v.25 n.4, p.329-337, July 1991
• 14: S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue. 1998. Galaxy-II: A reference architecture for conversational system development. In ICSLP-98.
• 15: Amanda Stent , John Dowding , Jean Mark Gawron , Elizabeth Owen Bratt , Robert Moore, The CommandTalk spoken dialogue system, Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, p.183-190, June 20-26, 1999, College Park, Maryland [doi>10.3115/1034678.1034713]
• 16: M. A. Walker, S. J. Whittaker, P. Maloor, J. D. Moore, M. Johnston, and G. Vasireddy. 2002. Speech-Plans: Generating evaluative responses in spoken dialogue. In In Proceedings of INLG-02.