Natural multimodal dialogue systems: a configurable dialogue and presentation strategies component

ABSTRACT

In the context of natural multimodal dialogue systems, we address the challenging issue of the definition of cooperative answers in an appropriate multimodal form. Highlighting the intertwined relation of multimodal outputs with the content, we focus on the Dialogic strategy component, a component that defines from the set of possible contents to answer a user's request, the content to be presented to the user and its multimodal presentation. The content selection and the presentation allocation managed by the Dialogic strategy component are based on various constraints such as the availability of a modality and the user's preferences. Considering three generic types of dialogue strategies and their corresponding handled types of information as well as three generic types of presentation tasks, we present a first implementation of the Dialogic strategy component based on rules. By providing a graphical interface to configure the component by editing the rules, we show how the component can be easily modified by ergonomists at design time for exploring different solutions. In further work we envision letting the user modify the component at runtime.

REFERENCES

• 1: Berton, A., Bühler, D. and Minker, W. SmartKom-mobile car: User interaction with mobile services in a car environment. In Wahlster, W. (ed.), SmartKom: foundations of multimodal dialogue systems, 523--537, Springer, 2006.
• 2: M. Bordegoni , G. Faconti , S. Feiner , M. T. Maybury , T. Rist , S. Ruggieri , P. Trahanias , M. Wilson, A standard reference model for intelligent multimedia presentation systems, Computer Standards & Interfaces, v.18 n.6-7, p.477-496, Dec. 1997 [doi>10.1016/S0920-5489(97)00013-5]
• 3: Bunt, H., Kipp, M., Maybury, M. and Wahlster, W. Fusion and coordination for multimodal interactive information presentation. In Stock, O. and Zancanaro, M. (ed.) Multimodal intelligent information presentation, 325--339, Springer, 2005.
• 4: Christian Elting , Stefan Rapp , Gregor Möhler , Michael Strube, Architecture and implementation of multimodal plug and play, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958453]
• 5: Fréard, D., Jamet, E., Le Bohec, O., Poulain, G. and Botherel, V. Subjective measurement of workload related to a multimodal interaction task: Nasa-tlx vs. Workload profile. In Proc. HCII'07, Springer, 2007.
• 6: Meriam Horchani , Laurence Nigay , Franck Panaget, A platform for output dialogic strategies in natural multimodal dialogue systems, Proceedings of the 12th international conference on Intelligent user interfaces, January 28-31, 2007, Honolulu, Hawaii, USA [doi>10.1145/1216295.1216334]
• 7: Horchani, M., Fréard, D., Jamet, E., Nigay, L. and Panaget, F. A platform for designing multimodal dialogic and presentation strategies. In Proc. of the 2007 Workshop on the Semantics and Pragmatics of Dialogue DECALOG, 169--170, 2007.
• 8: JADE, Java Agent DEvelopment Framework, jade.tilab.com.
• 9: Louis, V. and Martinez T. JADE semantics framework. In Bellifemine, F., Caire, G. and Greenwood, D. (ed.), Developping multi-agent systems with JADE, John Wiley & Sons, 2007.
• 10: Mansoux, B., Nigay, L., and Troccaz, J. Output Multimodal Interaction: The Case of Augmented Surgery. In Proc. of the 20th British Human Computer Interaction Group conference HCI'06, 177--192, BCS Conference Series, Springer-Verlag and ACM Press, 2006.
• 11: Steven F. Roth , William E Hefley, Intelligent multimedia presentation systems: research and principles, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 12: Cyril Rousseau , Yacine Bellik , Frédéric Vernier , Didier Bazalgette, A framework for the intelligent multimodal presentation of information, Signal Processing, v.86 n.12, p.3696-3713, December 2006 [doi>10.1016/j.sigpro.2006.02.041]
• 13: Sadek, D. Design considerations on dialogue systems: from theory to technology. In IDS'99, 173--187, 1999.
• 14: Anoop K. Sinha , James A. Landay, Capturing user tests in a multimodal, multidevice informal prototyping tool, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958457]
• 15: A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop, ACM SIGCHI Bulletin, v.24 n.1, p.32-37, Jan. 1992 [doi>10.1145/142394.142401]
• 16: Wahlster, W. Dialogue systems go multimodal: the SmartKom experience. In Wahlster, W. (ed.), SmartKom: foundations of multimodal dialogue systems, 3--27, Springer, 2006.
• 17: Michelle X. Zhou , Zhen Wen , Vikram Aggarwal, A graph-matching approach to dynamic media allocation in intelligent multimedia interfaces, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040861]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
          Subjects: User interfaces

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Prototyping; Natural language

General Terms:
Algorithms, Human Factors

Keywords:
development tool, dialogue and presentation strategies, multimodal output