Towards adaptive programming: integrating reinforcement learning into a programming language

ABSTRACT

Current programming languages and software engineering paradigms are proving insufficient for building intelligent multi-agent systems--such as interactive games and narratives--where developers are called upon to write increasingly complex behavior for agents in dynamic environments. A promising solution is to build adaptive systems; that is, to develop software written specifically to adapt to its environment by changing its behavior in response to what it observes in the world. In this paper we describe a new programming language, An Adaptive Behavior Language (A2BL), that implements adaptive programming primitives to support partial programming, a paradigm in which a programmer need only specify the details of behavior known at code-writing time, leaving the run-time system to learn the rest. Partial programming enables programmers to more easily encode software agents that are difficult to write in existing languages that do not offer language-level support for adaptivity. We motivate the use of partial programming with an example agent coded in a cutting-edge, but non-adaptive agent programming language (ABL), and show how A2BL can encode the same agent much more naturally.

REFERENCES

• 1: David Andre and Stuart Russell. Programmable reinforcement learning agents. In Advances in Neural Information Processing Systems, volume 13, 2001.
• 2: David Andre , Stuart J. Russell, State abstraction for programmable reinforcement learning agents, Eighteenth national conference on Artificial intelligence, p.119-125, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 3: Sooraj Bhat, Charles Isbell, and Michael Mateas. On the difficulty of modular reinforcement learning for real-world partial programming. In Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, MA, USA, July 2006.
• 4: Thomas G. Dietterich, The MAXQ Method for Hierarchical Reinforcement Learning, Proceedings of the Fifteenth International Conference on Machine Learning, p.118-126, July 24-27, 1998
• 5: Leslie Pack Kaelbling, Michael L. Littman, and Andrew P. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research}, 237--285, 1996.
• 6: A. B. Loyall and J. Bates. Hap: A reactive adaptive architecture for agents. Technical Report CMU-CS-91-147, 1991.
• 7: Michael Mateas and Andrew Stern. Facade: An experiment in building a fully-realized interactive drama. In Game Developers Conference: Game Design Track, San Jose, CA, March 2003.
• 8: Michael Mateas and Andrew Stern. Life-like Characters. Tools, Affective Functions and Applications, chapter A Behavior Language: Joint Action and Behavioral Idioms. Springer, 2004.
• 9: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 10: Peter Norvig. Decision theory: The language of adaptive agent software. Presentation, March 1998. http://www.norvig.com/adaptive/index.htm
• 11: Peter Norvig , David Cohn, Adaptive software, PC AI, v.11 n.1, p.27-30, Jan./Feb. 1997
• 12: Ronald Parr , Stuart Russell, Reinforcement learning with hierarchies of machines, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.1043-1049, July 1998, Denver, Colorado, United States
• 13: Stuart J. Russell , Peter Norvig, Artificial Intelligence: A Modern Approach, Pearson Education, 2003
• 14: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 15: Sprague, N., and Ballard, D. Multiple-Goal Reinforcement Learning with Modular Sarsa(0). In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, 2003. Workshop paper.

INDEX TERMS

Primary Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features

General Terms:
Algorithms, Design, Languages

Keywords:
adaptive programming, object-oriented programming, partial programming, reinforcement learning