Motivated reinforcement learning for adaptive characters in open-ended simulation games

ABSTRACT

Recently a new generation of virtual worlds has emerged in which users are provided with open-ended modelling tools with which they can create and modify world content. The result is evolving virtual spaces for commerce, education and social interaction. In general, these virtual worlds are not games and have no concept of winning, however the open-ended modelling capacity is nonetheless compelling. The rising popularity of open-ended virtual worlds suggests that there may also be potential for a new generation of computer games situated in open-ended environments. A key issue with the development of such games, however, is the design of non-player characters which can respond autonomously to unpredictable, open-ended changes to their environment. This paper considers the impact of open-ended modelling on character development in simulation games. Motivated reinforcement learning using context-free grammars is proposed as a means of representing unpredictable, evolving worlds for character reasoning. This technique is used to design adaptive characters for the Second Life virtual world to create a new kind of open-ended simulation game.

REFERENCES

• 1: Activeworlds. "Activeworlds". www.activeworlds.com (Accessed January 2007).
• 2: Johnson, D. and Wiles, J. "Computer games with intelligence", presented at The 10th IEEE International Conference on Fuzzy Systems, pp1355--1358, 2001.
• 3: John E. Laird , Michael van Lent, Human-Level AI's Killer Application: Interactive Computer Games, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.1171-1178, July 30-August 03, 2000
• 4: Linden. "Second Life", www.secondlife.com (accessed January, 2007).
• 5: Maher, M.-L. and Gero, J.S. "Agent models of 3D virtual worlds", ACADIA 2002: Thresholds. California State Polytechnic University, Pamona, pp. 127--138, 2002.
• 6: Merceron, A. Languages and Logic: Pearson Education Australia, 2001.
• 7: Kathryn Merrick , Mary Lou Maher, Motivated reinforcement learning for non-player characters in persistent computer game worlds, Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, June 14-16, 2006, Hollywood, California [doi>10.1145/1178823.1178828]
• 8: Merrick, K. 2007, "Modelling Motivation for Experience-Based Attention Focus in Reinforcement Learning", PhD Thesis, University of Sydney (manuscript).
• 9: Nilsson, N. J. Introduction to machine learning: http://ai.stanford.edu/people/nilsson/mlbook.html (accessed January, 2006), 1996.
• 10: Rex, F. "LambdaMOO: An introduction", http://www.lambdamoo.info, (accessed December, 2006).
• 11: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 12: Woodcock, S. "Games making interesting use of artificial intelligence techniques" http://www.gameai.com/games.html (accessed October 2005).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search
          Subjects: Dynamic programming

General Terms:
Algorithms

Keywords:
adaptive characters, computer games, context-free grammar, motivated reinforcement learning