Compelling experiences in mixed reality interactive storytelling

ABSTRACT

Entertainment systems promise to be a significant application for Mixed Reality. Recently, a growing number of Mixed Reality applications have included interaction with virtual actors and storytelling. However, AI-based Interactive Storytelling techniques have not yet been explored in the context of Mixed Reality. In this paper, we describe a fully-integrated first prototype based upon the adaptation of an Interactive Storytelling technique to a Mixed Reality system. After a description of the real time image processing techniques that support the creation of a hybrid environment, we introduce the storytelling technique and the essential modalities of user interactions in the Mixed Reality context. Finally, we illustrate these experiments by discussing examples obtained from the system.

REFERENCES

• 1: A. F. Bobick, S. S. Intille, J. W. Davis, F. Baird, C. S. Pinhanez, L. W. Campbell, Y. A. Ivanov, A. Schutte, and Andrew Wilson, "The kidsroom: A perceptually-based interactive and immersive story environment", Technical Report 398, MIT Media Lab Perceptual Computer Section, 1996.
• 2: R. Barthes, "Introduction à l'Analyse Structurale des Récits (in French). Communications, 8, 1966, pp. 1--27.
• 3: B. MacIntyre, J. D. Bolter, J. Vaughn, B. Hannigan, M. Gandy, E. Moreno, M. Haas, S. H. Kang, D. Krum and S. Voida, "Three Angry Men: An Augmented-Reality Experiment in Point-of-View Drama." In Proceedings of TIDSE 2003, Darmstadt, Germany, March 24--26, 2003.
• 4: Jay David Bolter , Richard Grusin, Remediation: understanding new media, MIT Press, Cambridge, MA, 1998
• 5: Marc Cavazza , Fred Charles , Steven J. Mead, Character-Based Interactive Storytelling, IEEE Intelligent Systems, v.17 n.4, p.17-24, July 2002 [doi>10.1109/MIS.2002.1024747]
• 6: Marc Cavazza , Fred Charles , Steven J. Mead, Interacting with virtual characters in interactive storytelling, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 1, July 15-19, 2002, Bologna, Italy [doi>10.1145/544741.544819]
• 7: F. Charles, J. L. Lugrin, M. Cavazza, and S. J. Mead, "Real-Time Camera Control for Interactive Storytelling", GameOn 2002, London, UK, 2002.
• 8: Adrian David Cheok , Wang Weihua , Xubo Yang , Simon Prince , Fong Siew Wan , Mark Billinghurst , Hirokazu Kato, Interactive Theatre Experience in Embodied + Wearable Mixed Reality Space, Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR'02), p.317, September 30-October 01, 2002
• 9: Martin Faust, Mixed reality gaming environment QuakeRunner, Proceedings of the second international conference on Entertainment computing, p.1-4, May 08-10, 2003, Pittsburgh, Pennsylvania
• 10: C. Geib, "Intentions in means-end planning", Technical Report MC-CIS-92-73, University of Pennsylvania, 1992.
• 11: Pattie Maes , Trevor Darrell , Bruce Blumberg , Alex Pentland, The ALIVE system: wireless, full-body interaction with autonomous agents, Multimedia Systems, v.5 n.2, p.105-112, March 1997 [doi>10.1007/s005300050046]
• 12: Jeffrey Jacobson , Zimmy Hwang, Unreal tournament for immersive interactive theater, Communications of the ACM, v.45 n.1, January 2002 [doi>10.1145/502269.502292]
• 13: X. Marichal, and T. Umeda, "Real-Time Segmentation of Video Objects for Mixed-Reality Interactive Applications", Proceedings of the "SPIE's Visual Communications and Image Processing" (VCIP 2003) International Conference, Lugano, Switzerland, 2003.
• 14: Michael Mateas , Andrew Stern, A Behavior Language for Story-Based Believable Agents, IEEE Intelligent Systems, v.17 n.4, p.39-47, July 2002 [doi>10.1109/MIS.2002.1024751]
• 15: M. Mateas, "An Oz-Centric Review of Interactive Drama and Believable Agents", Technical Report CMUCS-97-156, Department of Computer Science, Carnegie Mellon University, Pittsburgh, USA, 1997.
• 16: R. Nakatsu, and N. Tosa, "Active Immersion: The Goal of the Communications with Interactive Agents", Autonomous Agents 2000 Workshop on Achieving Human-Like Behavior in Interactive Animated Agents, Barcelona, Spain, 2000.
• 17: A. Nandi, and X. Marichal, "Senses of Spaces through Transfiction", pp. 439--446 in "Entertainment Computing: Technologies and Applications" (Proceedings of the International Workshop on Entertainment Computing, IWEC 2002), R. Nakatsu and J. Hoshino (eds), Kluwer Academic Publishers, 2003, ISBN 1-4020-7360-7
• 18: Dana S. Nau , Yue Cao , Amnon Lotem , Hector Muñoz-Avila, SHOP: Simple Hierarchical Ordered Planner, Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, p.968-975, July 31-August 06, 1999
• 19: Jun Ohya , Akira Utsumi , Junji Yamato, Analyzing Video Sequences of Multiple Humans, Kluwer Academic Publishers, Norwell, MA, 2002
• 20: K. Perlin, and A. Goldberg, "Improv: A System for Scripting Interactive Actors in Virtual Worlds", Computer Graphics; Vol. 29 No. 3, 1996.
• 21: Wayne Piekarski , Bruce Thomas, ARQuake: the outdoor augmented reality gaming system, Communications of the ACM, v.45 n.1, January 2002 [doi>10.1145/502269.502291]
• 22: Mark Riedl , C. J. Saretto , R. Michael Young, Managing interaction between users and agents in a multi-agent storytelling environment, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860694]
• 23: E. Salvador, A. Cavallaro, and T. Ebrahimi, "Shadow identification and classification using invariant color models", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP2001), Salt Lake City (USA), 2001.
• 24: N. M. Sgouros, G. Papakonstantinou and P. Tsanakas, "A Framework for Plot Con-trol in Interactive Story Systems", Proceedings AAAI'96, Portland, AAAI Press, 1996.
• 25: Christopher Stapleton , Charles Hughes , Michael Moshell , Paulius Micikevicius , Marty Altman, Applying Mixed Reality to Entertainment, Computer, v.35 n.12, p.122-124, December 2002 [doi>10.1109/MC.2002.1106186]
• 26: R. Hill , J. Gratch , W. L. Johnson , C. Kyriakakis , C. LaBore , R. Lindheim , S. Marsella , D. Miraglia , B. Moore , J. Morie , J. Rickel , M. Thiébaux , L. Tuch , R. Whitney , J. Douglas , W. Swartout, Toward the holodeck: integrating graphics, sound, character and story, Proceedings of the fifth international conference on Autonomous agents, p.409-416, May 2001, Montreal, Quebec, Canada [doi>10.1145/375735.376390]
• 27: R. M. Young, "Creating Interactive Narrative Structures: The Potential for AI Approaches", AAAI Spring Symposium in Artificial Intelligence and Interactive Entertainment, AAAI Press, 2000.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Intelligent agents

General Terms:
Algorithms, Design, Experimentation, Human Factors, Languages

Keywords:
interactive storytelling, mixed reality, virtual actors