A task specification language for bootstrap learning

ABSTRACT

Traditionally, research in the reinforcement learning (RL) community has been devoted to developing domain-independent algorithms such as SARSA [13], Q-learning [16], prioritized sweeping [8], or LSPI [6], that are designed to work for any given state space and action space. However, the modus operandi in RL research has been for a human expert to re-code each learning environment, including defining the actions and state features, as well as specifying the algorithm to be used. Typically each new RL experiment is run by explicitly calling a new program (even when learning can be biased by previous learning experiences, as in transfer learning [10, 15, 14]). Thus, while standards have developed for describing and testing individual RL algorithms (e.g., RL-Glue [17]), no such standards have developed for the problem of describing complete tasks to a preexisting agent.

REFERENCES

• 1: Bootstrapped learning proposer information pamphlet. http://www.darpa.mil/IPTO/solicit/closed/BAA-07-04_PIP.pdf.
• 2: Peter Dayan , Geoffrey E. Hinton, Feudal Reinforcement Learning, Advances in Neural Information Processing Systems 5, [NIPS Conference], p.271-278, November 30-December 03, 1992
• 3: Thomas G. Dietterich, The MAXQ Method for Hierarchical Reinforcement Learning, Proceedings of the Fifteenth International Conference on Machine Learning, p.118-126, July 24-27, 1998
• 4: T. Jung and D. Polani. Learning robocup-keepaway with kernels. In JMLR Workshop and Conference Proceedings ("Gaussian Processes in Practice"), volume 1. JMLR, 2007.
• 5: G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language advice: Initial results in robocup soccer. In AAAI-2004 Workshop on Supervisory Control of Learning and Adaptive Systems, 2004.
• 6: Michail G. Lagoudakis , Ronald Parr, Least-squares policy iteration, The Journal of Machine Learning Research, 4, p.1107-1149, 12/1/2003
• 7: Richard Maclin , Jude W. Shavlik, Creating advice-taking reinforcement learners, Machine Learning, v.22 n.1-3, p.251-281, Jan./Feb./March 1996 [doi>10.1007/BF00114730]
• 8: Andrew W. Moore , Christopher G. Atkeson, Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time, Machine Learning, v.13 n.1, p.103-130, Oct. 1993 [doi>10.1023/A:1022635613229]
• 9: S. Schaal. Learning from demonstration. NIPS 9, 1997.
• 10: Vishal Soni , Satinder Singh, Using Homomorphisms to transfer options across continuous reinforcement learning domains, Proceedings of the 21st national conference on Artificial intelligence, p.494-499, July 16-20, 2006, Boston, Massachusetts
• 11: P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for RoboCup-soccer keepaway. Adaptive Behavior, 13(3):165--188, 2005.
• 12: Richard S. Sutton , Doina Precup , Satinder Singh, Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning, Artificial Intelligence, v.112 n.1-2, p.181-211, Aug. 1999 [doi>10.1016/S0004-3702(99)00052-1]
• 13: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 14: Matthew E. Taylor , Peter Stone, Behavior transfer for value-function-based reinforcement learning, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082482]
• 15: L. Torrey, T. Walker, J. Shavlik, and R. Maclin. Using advice to transfer knowledge acquired in one reinforcement learning task to another. In ECML 2005. Porto, Portugal, 2005.
• 16: C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, UK, 1989.
• 17: A. White, M. Lee, A. Butcher, B. Tanner, L. Hackman, and R. Sutton. Rl-glue distribution, http://rlai.cs.ualberta.ca/rlbb/top.html, 2007.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Intelligent agents

General Terms:
Human Factors, Languages, Standardization

Keywords:
human computer interaction, reinforcement learning