An architecture to provide adaptive, synchronized and multimodal human computer interaction

ABSTRACT

In this paper we present a solution to compose synchronized multimodal computer human interaction with a pattern-oriented approach. The focus of this paper is how to synchronize the different modalities of interaction.In our application scenario the system consists of mobile agents and the user interface is generated from a dialog description language, which is XML-encoded. The generated User Interface is executed by one or more UI-engines, which can use one or more modalities like a text chat system, a speech-based system or a graphical, window-oriented user interface. The user interface is separated and also structured by so-called dialog moves which are suitable to be adapted to different UI-engines. The system supports more than one UI-engine at the same time which results in a possibly multimodal interaction. The interaction is synchronized by observing the events produced by the dialog moves of each UI-engine.The system is evaluated in a project called MAP¹ which deals with new human computer interaction methods in future mobile work environments.

REFERENCES

• 1: Bray, T., et al. Extensible Markup Language (XML). W3C Proposed Recommendation 10-February-1998, REC-xml-19980210, February 10, 1998, <u>http://www.w3.org/TR/REC-xml</u>.
• 2: Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., and Young, R. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE properties. In Proceedings of INTERACT'95, Lillehammer, Norway, June 1995.
• 3: Faure, C., and Julia, L. An Agent-Based Architecture for a Multimodal Interface. Working notes of the AAAI Intelligent Multi-Media Multi-Modal Systems Symp. Standford Univ., 21--23 march, 1994.
• 4: Erich Gamma , Richard Helm , Ralph Johnson , John Vlissides, Design patterns: elements of reusable object-oriented software, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 5: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 6: Kuhlmann, H. Multimedia Workplace of the Future. Office, e-Business and e-Work Conference (e2001), Venice, Italy, October 2001
• 7: Laurence Nigay , Joëlle Coutaz, A generic platform for addressing the multimodal challenge, Proceedings of the SIGCHI conference on Human factors in computing systems, p.98-105, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223917]
• 8: Nigay, L., and Coutaz, J. Multifeature systems: From HCI properties to software design. In IMMI 95, July 1995
• 9: A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop, ACM SIGCHI Bulletin, v.24 n.1, p.32-37, Jan. 1992 [doi>10.1145/142394.142401]
• 10: Minh Tue Vo , Alex Waibel, A framework and toolkit for the construction of multimodal learning interfaces, 1998
• 11: Vo, M.T., and Waibel, A. Modeling and interpreting multimodal inputs: A semantic integration approach. Technical Report CMU-CS-97-192. Carnegie Mellon University, 1997.

INDEX TERMS

Keywords:
adaptive, dialog move, mobile agents, multimodal, synchronisation