Evaluation of spoken multimodal conversation

ABSTRACT

Spoken multimodal dialogue systems in which users address face-only or embodied interface agents have been gaining ground in research for some time. Although most systems are still strictly task-oriented, the field is now moving towards domain-oriented systems and real conversational systems which are no longer defined in terms of the task(s) they support. This paper describes the first running prototype of such a system which enables spoken and gesture interaction with life-like fairytale author Hans Christian Andersen about his fairytales, life, study, etc., focusing on multimodal conversation. We then present recent user test evaluation results on multimodal conversation.

REFERENCES

• 1: Bernsen, N. O.: When H. C. Andersen is not talking back In Rist, T., Aylet, R., Ballin, D. and Rickel, J. (Eds.): Proc. of the Fourth International Working Conference on Intelligent Virtual Agents (IVA'2003). Berlin: Springer Verlag 2003, 27--30.
• 2: Niels Ole Bernsen , Marcela Charfuelàn , Andrea Corradini , Laila Dybkjær , Thomas Hansen , Svend Kiilerich , Mykola Kolodnytsky , Dmytro Kupkin , Manish Mehta, First prototype of conversational H.C. Andersen, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989951]
• 3: Bernsen, N. O., Dybkjaer, H. and Dybkjaer, L.: Cooperativity in human-machine and human-human spoken dialogue. Discourse Processes, Vol. 21, No. 2, 1996, 213--236.
• 4: Bernsen, N. O. and Dybkjaer, L.: Domain-Oriented Conversation with H.C. Andersen. Proc. of the Workshop on Affective Dialogue Systems, Kloster Irsee, Germany, June 2004. Berlin: Springer Verlag: Lecture Notes in Artificial Intelligence, Vol. 3068, 2004, 142--153.
• 5: Bernsen, N. O., Dybkjaer, L. and Kiilerich, S.: Evaluating Conversation with Hans Christian Andersen. Proc. of the Fourth International Conference on Language Resources and Evaluation (LREC), Lisbon, Portugal, May 2004, Vol. 3, 2004, 1011--1014.
• 6: Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (Eds.): Embodied conversational agents. Cambridge, MS: MIT Press 2000.
• 7: Clark, H.H.: Using Language. Cambridge: Cambridge University Press, 1996.
• 8: Cole, R., van Vuuren, S., Pellom, B., Hacioglu, K., Ma, J., Movellan, J., Schwartz, S., Wade-Stein, D., Ward, W. and Yan, J.: Perceptive Animated Interfaces: First Steps towards a New Paradigm for Human-Computer Interaction. In van Kuppevelt, J., Dybkjær, L. and Bernsen, N.O. (Eds.): Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Dordrecht: Kluwer, 2004 (to appear).
• 9: Corradini, A., Fredriksson, M., Mehta, M., Königsmann, J., Bernsen, N. O., and Johannesson, L.: Towards Believable Behavior Generation for Embodied Conversational Agents. Proc. of the Workshop on Interactive Visualisation and Interaction Technologies, IV&IT, Krakow, Poland, June 2004. Berlin: Springer Verlag: Lecture Notes in Computer Science, Vol. 3038, 2004, 946--953.
• 10: Dybkjær, L., Bernsen, N.O. and Minker, W.: Evaluation and Usability of Multimodal Spoken Language Dialogue Systems. Speech Communication, Vol. 43, 2004, 33--54.
• 11: Gabor, D.: How to Start a Conversation and Make Friends. London: Sheldon Press 2001.
• 12: Grice, P.: Logic and conversation. In P. Cole and J.L. Morgan (Eds.): Syntax and semantics Vol. 3: Speech acts (41-58). New York: Academic Press 1975. Reprinted in Grice, P.: Studies in the way of words. Cambridge, MA: Harvard University Press 1989.
• 13: Gustafson, J., Bell, L., Lindström, A. and Wirén, M.: The NICE Fairy-tale Game System. Proc. of 5th SIGdial Workshop on Discourse and Dialogue, Cambridge, MA, 2004.
• 14: Gustafson, J., Lindberg, N. and Lundeberg, M.: The August Spoken Dialogue System. Proc. of Eurospeech, 1999, 1151--1154.
• 15: Turing, A.: Computing machinery and intelligence. Mind 59, 1950, 433--60.
• 16: ISI: http://www.isi.edu/isd/carte/proj_tactlang/
• 17: KTH broker: http://www.speech.kth.se/broker
• 18: NICE: http://www.niceproject.com/

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Animations

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Audio input/output; Artificial, augmented, and virtual realities; Evaluation/methodology
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI); Evaluation/methodology; Natural language; Voice I/O

General Terms:
Design, Experimentation, Human Factors, Management, Measurement, Performance

Keywords:
evaluation, natural interaction, spoken conversation