Contextual information access and storytelling in mixed reality using hypermedia

ABSTRACT

This article describes gaming and storytelling activities in a mixed environment that integrates the real and virtual worlds, uses an augmented reality paradigm, and is supported by a structuring and presentation framework for use in context-aware mixed-reality applications. The basis of the framework is a generic hypermedia model that can handle different media elements, objects, and relations between spaces and locations in physical and virtual worlds. A main component of the model deals with providing contextual content according to the state of the application and the person using it. A storytelling layer was also defined, mainly by using the contextual mechanisms of the base model. This layer provides abstractions to storytelling applications that reflect the morphology of common story structures and supports gaming flow. The framework is being tested in a gaming and storytelling environment that integrates the real world, media elements, and virtual 3D worlds.

REFERENCES

• 1: Ronald Azuma , Yohan Baillot , Reinhold Behringer , Steven Feiner , Simon Julier , Blair MacIntyre, Recent Advances in Augmented Reality, IEEE Computer Graphics and Applications, v.21 n.6, p.34-47, November 2001 [doi>10.1109/38.963459]
• 2: Jörg Baus , Antonio Krüger , Wolfgang Wahlster, A resource-adaptive mobile navigation system, Proceedings of the 7th international conference on Intelligent user interfaces, January 13-16, 2002, San Francisco, California, USA [doi>10.1145/502716.502723]
• 3: Francesco Bellotti , Riccardo Berta , Alessandro De Gloria , Massimiliano Margarone, User Testing a Hypermedia Tour Guide, IEEE Pervasive Computing, v.1 n.2, p.33-41, April 2002 [doi>10.1109/MPRV.2002.1012335]
• 4: Mark Billinghurst , Hirkazu Kato , Ivan Poupyrev, The MagicBook—Moving Seamlessly between Reality and Virtuality, IEEE Computer Graphics and Applications, v.21 n.3, p.6-8, May 2001
• 5: Bjork, S., Falk, J., Hanson, R., and Ljungstrand, P. 2001. Pirates! Using the physical world as a game board. INTERACT (2001).
• 6: Branigan, E. 1992. Narrative comprehension and film. In Sightlines, vol. 1. E. Buscombe. ed. Routledge, London, 1992.
• 7: Kevin M. Brooks, Do story agents use rocking chairs? The theory and implementation of one model for computational narrative, Proceedings of the fourth ACM international conference on Multimedia, p.317-328, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244233]
• 8: Nolan Bushnell, Relationships between fun and the computer business, Communications of the ACM, v.39 n.8, p.31-37, Aug. 1996 [doi>10.1145/232014.232025]
• 9: Brad Campbell , Joseph M. Goodman, HAM: a general-purpose hypertext abstract machine, Proceeding of the ACM conference on Hypertext, p.21-32, November 1987, Chapel Hill, North Carolina, United States [doi>10.1145/317426.317429]
• 10: Keith Cheverst , Keith Mitchell , Nigel Davies, The role of adaptive hypermedia in a context-aware tourist GUIDE, Communications of the ACM, v.45 n.5, May 2002 [doi>10.1145/506218.506244]
• 11: Cooper, S. 2001 Scripting for interactive narrative. In Contents Drive Technology. 2001, 102-117.
• 12: Paul De Bra , Geert-Jan Houben , Hongjing Wu, AHAM: a Dexter-based reference model for adaptive hypermedia, Proceedings of the tenth ACM Conference on Hypertext and hypermedia : returning to our diverse roots: returning to our diverse roots, p.147-156, February 21-25, 1999, Darmstadt, Germany [doi>10.1145/294469.294508]
• 13: Norman M. Delisle , Mayer D. Schwartz, Contexts—a partitioning concept for hypertext, ACM Transactions on Information Systems (TOIS), v.5 n.2, p.168-186, April 1987 [doi>10.1145/27636.27639]
• 14: Sastry Duri , Alan Cole , Jonathan Munson , Jim Christensen, An approach to providing a seamless end-user experience for location-aware applications, Proceedings of the 1st international workshop on Mobile commerce, p.20-25, July 2001, Rome, Italy [doi>10.1145/381461.381465]
• 15: Kaj Grønbæk , Randall H. Trigg, Design issues for a Dexter-based hypermedia system, Proceedings of the ACM conference on Hypertext, p.191-200, November 30-December 04, 1992, Milan, Italy [doi>10.1145/168466.168525]
• 16: Kaj Grønbæk , Peter Posselt Vestergaard , Peter Ørbæk, Towards geo-spatial hypermedia: Concepts and prototype implementation, Proceedings of the thirteenth ACM conference on Hypertext and hypermedia, June 11-15, 2002, College Park, Maryland, USA [doi>10.1145/513338.513370]
• 17: Frank G. Halasz , Thomas P. Moran , Randall H. Trigg, Notecards in a nutshell, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.45-52, April 05-09, 1987, Toronto, Ontario, Canada
• 18: Frank Halasz , Mayer Schwartz, The Dexter hypertext reference model, Communications of the ACM, v.37 n.2, p.30-39, Feb. 1994 [doi>10.1145/175235.175237]
• 19: Tony Hall , Luigina Ciolfi , Liam Bannon , Mike Fraser , Steve Benford , John Bowers , Chris Greenhalgh , Sten-Olof Hellström , Shahram Izadi , Holger Schnädelbach , Martin Flintham, The visitor as virtual archaeologist: explorations in mixed reality technology to enhance educational and social interaction in the museum, Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, November 28-30, 2001, Glyfada, Greece [doi>10.1145/584993.585008]
• 20: Lynda Hardman , Dick C. A. Bulterman , Guido van Rossum, The Amsterdam hypermedia model: adding time and context to the Dexter model, Communications of the ACM, v.37 n.2, p.50-62, Feb. 1994 [doi>10.1145/175235.175239]
• 21: Korkea-aho, M. 2000. Context-aware applications survey. Tech. Rep., Dept of Computer Science, Helsinki Univ. of Technology, April 2000. M. http://www.hut.fi/~mkorkeaa/doc/contextaware. html.
• 22: Ursula Kretschmer , Volker Coors , Ulrike Spierling , Dieter Grasbon , Kerstin Schneider , Isabel Rojas , Rainer Malaka, Meeting the spirit of history, Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, November 28-30, 2001, Glyfada, Greece [doi>10.1145/584993.585016]
• 23: Sue Long , Rob Kooper , Gregory D. Abowd , Christopher G. Atkeson, Rapid prototyping of mobile context-aware applications: the Cyberguide case study, Proceedings of the 2nd annual international conference on Mobile computing and networking, p.97-107, November 1996, Rye, New York, United States [doi>10.1145/236387.236412]
• 24: Mateas, M. and Stern, A. 2003. Integrating plot, character and natural language processing in the interactive drama. In Proceedings of the Conference on Technologies for Interactive Digital Storytelling and Entertainment (2003). 139-151.
• 25: Norman Meyrowitz, Intermedia: The architecture and construction of an object-oriented hypemedia system and applications framework, Conference proceedings on Object-oriented programming systems, languages and applications, p.186-201, September 29-October 02, 1986, Portland, Oregon, United States
• 26: Pengkai Pan , Carly Kastner , David Crow , Glorianna Davenport, M-Studio: an authoring application for context-aware multimedia, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641082]
• 27: Wayne Piekarski , Bruce Thomas, ARQuake: the outdoor augmented reality gaming system, Communications of the ACM, v.45 n.1, January 2002 [doi>10.1145/502269.502291]
• 28: Propp, V. 1968. Morphology of the Folktale. L.A. Wagner, ed., University of Texas Press, Austin, 1968.
• 29: Teresa Romão , Eduardo Dias , José Danado , Nuno Correia , Adelaide Trabuco , Carlos Santos , Rossana Santos , Edmundo Nobre , António Câmara , Luís Romero, Augmenting reality with geo-referenced information for environmental management, Proceedings of the 10th ACM international symposium on Advances in geographic information systems, November 08-09, 2002, McLean, Virginia, USA [doi>10.1145/585147.585184]
• 30: Luis Romero , Nuno Correia, HyperReal: a hypermedia model for mixed reality, Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, August 26-30, 2003, Nottingham, UK [doi>10.1145/900051.900055]
• 31: Jorge Santiago , Luís Romero , Nuno Correia, A mixed reality mystery game, Proceedings of the second international conference on Entertainment computing, p.1-8, May 08-10, 2003, Pittsburgh, Pennsylvania
• 32: Nitin Sawhney , David Balcom , Ian Smith, HyperCafe: narrative and aesthetic properties of hypervideo, Proceedings of the the seventh ACM conference on Hypertext, p.1-10, March 16-20, 1996, Bethesda, Maryland, United States [doi>10.1145/234828.234829]
• 33: Thad Starner , Bastian Leibe , Brad Singletary , Jarrell Pair, MIND-WARPING: towards creating a compelling collaborative augmented reality game, Proceedings of the 5th international conference on Intelligent user interfaces, p.256-259, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325864]
• 34: Steed, A., Maccoll, I., Randell, C., and Brown, B. et al. 2002. The use of spatial models in mixed-reality systems (draft). Tech. Rep. Equator-02-040, UCL, Glasgow, Oct. 2002.
• 35: Zsolt Szalavári , Erik Eckstein , Michael Gervautz, Collaborative gaming in augmented reality, Proceedings of the ACM symposium on Virtual reality software and technology, p.195-204, November 02-05, 1998, Taipei, Taiwan [doi>10.1145/293701.293740]
• 36: Hideyuki Tamura , Hiroyuki Yamamoto , Akihiro Katayama, Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds, IEEE Computer Graphics and Applications, v.21 n.6, p.64-70, November 2001 [doi>10.1109/38.963462]
• 37: Bruce Thomas , Ben Close , John Donoghue , John Squires , Phillip de Bondi , Michael Morris , Wayne Piekarski, ARQuake: An Outdoor/Indoor Augmented Reality First Person Application, Proceedings of the 4th IEEE International Symposium on Wearable Computers, p.139, October 18-21, 2000
• 38: Wichert, R. 2002. Collaborative gaming in a mobile augmented reality environment. In Proceedings of First Ibero-American Symposium in Computer Graphics (SIACG, 2002).

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

General Terms:
Design, Experimentation, Human Factors, Theory

Keywords:
hypermedia interfaces, hypermedia model, mixed and augmented reality, mobile gaming and storytelling