Emotional input for character-based interactive storytelling

ABSTRACT

In most Interactive Storytelling systems, user interaction is based on natural language communication with virtual agents, either through isolated utterances or through dialogue. Natural language communication is also an essential element of interactive narratives in which the user is supposed to impersonate one of the story's characters. Whilst techniques for narrative generation and agent behaviour have made significant progress in recent years, natural language processing remains a bottleneck hampering the scalability of Interactive Storytelling systems. In this paper, we introduce a novel interaction technique based solely on emotional speech recognition. It allows the user to take part in dialogue with virtual actors without any constraints on style or expressivity, by mapping the recognised emotional categories to narrative situations and virtual characters feelings. Our Interactive Storytelling system uses an emotional planner to drive characters' behaviours. The main feature of this approach is that characters' feelings are part of the planning domain and are at the heart of narrative representations. The emotional speech recogniser analyses the speech signal to produce a variety of features which can be used to define ad-hoc categories on which to train the system. The content of our interactive narrative is an adaptation of one chapter of the XIX^th century classic novel, Madame Bovary, which is well suited to a formalisation in terms of characters' feelings. At various stages of the narrative, the user can address the main character or respond to her, impersonating her lover. The emotional category extracted from the user utterance can be analysed in terms of the current narrative context, which includes characters' beliefs, feelings and expectations, to produce a specific influence on the target character, which will become visible through a change in its behaviour, achieving a high level of realism for the interaction. A limited number of emotional categories is sufficient to drive the narrative across multiple courses of actions, since it comprises over thirty narrative functions. We report results from a fully implemented prototype, both in terms of proof of concept and of usability through a preliminary user study.

REFERENCES

• 1: Blai Bonet , Hector Geffner, Planning as Heuristic Search: New Results, Proceedings of the 5th European Conference on Planning: Recent Advances in AI Planning, p.360-372, September 08-10, 1999
• 2: Blai Bonet , Héctor Geffner, Planning as heuristic search, Artificial Intelligence, v.129 n.1-2, p.5-33, June 2001 [doi>10.1016/S0004-3702(01)00108-4]
• 3: Marc Cavazza , Fred Charles , Steven J. Mead, Character-Based Interactive Storytelling, IEEE Intelligent Systems, v.17 n.4, p.17-24, July 2002 [doi>10.1109/MIS.2002.1024747]
• 4: Chang, C.-C. and Lin, C.-J. 2001. LIB-SVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
• 5: Diener, H. and Oertel, K. 2006. Experimental Approach to Affective Interaction in Games. Edutainment 2006: pp. 507--518.
• 6: Flaubert, G. 1856. Madame Bovary. La revue de Paris (Ed.), France. (in French).
• 7: David Freeman, Creating Emotion in Games, New Riders Publishing, Thousand Oaks, CA, 2003
• 8: Gratch, J. Why you should buy an emotional planner, Proceedings of the Autonomous Agents 1999 Workshop on Emotion-based Agent Architectures (EBAA'99).
• 9: Christian Jones , Jamie Sutherland, Acoustic Emotion Recognition for Affective Computer Gaming, Affect and Emotion in Human-Computer Interaction: From Theory to Applications, Springer-Verlag, Berlin, Heidelberg, 2008 [doi>10.1007/978-3-540-85099-1_18]
• 10: Patrick Kenny , Thomas D. Parsons , Jonathan Gratch , Albert A. Rizzo, Evaluation of Justina: A Virtual Patient with PTSD, Proceedings of the 8th international conference on Intelligent Virtual Agents, September 01-03, 2008, Tokyo, Japan [doi>10.1007/978-3-540-85483-8_40]
• 11: Richard E. Korf, Real-time heuristic search, Artificial Intelligence, v.42 n.2-3, p.189-211, March 1990 [doi>10.1016/0004-3702(90)90054-4]
• 12: Leclerc, Y (Ed.). Plans et Scenarios de Madame Bovary, CNRS Editions, France, 1995 (in French).
• 13: Yaxin Liu , Sven Koenig , David Furcy, Speeding up the calculation of heuristics for heuristic search-based planning, Eighteenth national conference on Artificial intelligence, p.484-491, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 14: Louchart, S., Romano, D., M., Aylett, R., and Pickering, J. 2004. Speaking and acting - interacting language and action for an expressive character. In Proceedings for the AISB workshop, University of Leeds, UK.
• 15: Lyons, G., Sharma, P., Baker, M., O'Malley, S., and Shanahan, A. 2003. A computer gamebased EMG biofeedback system for muscle rehabilitation. In: Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE. volume 2. S. 1625--1628. Sept. 2003.
• 16: Stacy Marsella , Jonathan Gratch, A step toward irrationality: using emotion to change belief, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 1, July 15-19, 2002, Bologna, Italy [doi>10.1145/544741.544821]
• 17: Stacy Marsella , Jonathan Gratch, Modeling coping behavior in virtual humans: don't worry, be happy, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860626]
• 18: Mateas, M., Stern, A., 2004. Natural Language Understanding in Façade: Surface-Text Processing. TIDSE 2004: 3--13.
• 19: Mohamad, Y., Velasco, C., A, Berlage, T. 2003. An Approach toward building of adaptive training and therapeutic systems considering the users affective state. HCI International Proceedings 2003, Crete, Greece.
• 20: Janet Horowitz Murray, Hamlet on the Holodeck: The Future of Narrative in Cyberspace, MIT Press, Cambridge, MA, 1998
• 21: Nazir, A., Lim, M., Y., Kriegel, M., Aylett, R., Cawsey, A., Enz, S., Rizzo, P. and Hall, L. 2008. ORIENT: An Inter-Cultural Role-Play Game. In Narrative in Interactive Learning Environments NILE 2008 Conference.
• 22: Ana Paiva , Ricardo Chaves , Moisés Piedade , Adrian Bullock , Gerd Andersson , Kristina Höök, SenToy: a tangible interface to control the emotions of a synthetic character, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860809]
• 23: David Pizzi , Fred Charles , Jean-Luc Lugrin , Marc Cavazza, Interactive Storytelling with Literary Feelings, Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, September 12-14, 2007, Lisbon, Portugal [doi>10.1007/978-3-540-74889-2_55]
• 24: Rank, S., Petta, P. and Trappl R. 2006. Features of Emotional Planning in Software Agents, in Dellariccia G., et al.(eds.), Decision Theory and Multi-Agent Planning, Springer Wien/New York, 2006.
• 25: Matthias Rehm , Thurid Vogt , Michael Wissner , Nikolaus Bee, Dancing the night away: controlling a virtual karaoke dancer by multimodal expressive cues, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal
• 26: Mark Owen Riedl , R. Michael Young, An Intent-Driven Planner for Multi-Agent Story Generation, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, p.186-193, July 19-23, 2004, New York, New York [doi>10.1109/AAMAS.2004.63]
• 27: William Swartout , Jonathan Gratch , Randall W. Hill , Eduard Hovy , Stacy Marsella , Jeff Rickel , David Traum, Toward virtual humans, AI Magazine, v.27 n.2, p.96-108, July 2006
• 28: David Traum , Jeff Rickel , Jonathan Gratch , Stacy Marsella, Negotiation over tasks in hybrid human-agent teams for simulation-based training, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860646]
• 29: Velten, E. 1968. A laboratory task for induction of mood states. Behavior Research&Therapy, (6):473--482.
• 30: Thurid Vogt , Elisabeth André , Nikolaus Bee, EmoVoice -- A Framework for Online Recognition of Emotions from Voice, Proceedings of the 4th IEEE tutorial and research workshop on Perception and Interactive Technologies for Speech-Based Systems: Perception in Multimodal Dialogue Systems, June 16-18, 2008, Kloster Irsee, Germany [doi>10.1007/978-3-540-69369-7_21]
• 31: Weiß, S., Müller, W., Spierling, U., Steimle, F., 2005. Scenejo - An Interactive Storytelling Platform. In: Virtual Storytelling, Using Virtual Reality Technologies for Storytelling, Third International Conference, Proceedings, Strasbourg, France, pp. 77--80.
• 32: Wilting, J., Krahmer, E., and Swerts, M. 2006. Real vs. acted emotional speech. In Proceedings of Interspeech 2006 | ICSLP, Pittsburgh, PA, USA.
• 33: Young, R., M. 2000. Creating Interactive Narrative Structures: The Potential for AI Approaches. AAAI Spring Symposium in Artificial Intelligence and Entertainment. AAAI Press, 2000.
• 34: Young, R., M. 2001. An Overview of the Mimesis Architecture: Integrating Intelligent Narrative Control into an Existing Gaming Environment, The Working Notes of the AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, Stanford, CA, March 2001.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Algorithms, Human Factors

Keywords:
affective interfaces, embodied conversational agents, interactive narrative