Autonomous transfer for reinforcement learning

ABSTRACT

Recent work in transfer learning has succeeded in making reinforcement learning algorithms more efficient by incorporating knowledge from previous tasks. However, such methods typically must be provided either a full model of the tasks or an explicit relation mapping one task into the other. An autonomous agent may not have access to such high-level information, but would be able to analyze its experience to find similarities between tasks. In this paper we introduce Modeling Approximate State Transitions by Exploiting Regression (MASTER), a method for automatically learning a mapping from one task to another through an agent's experience. We empirically demonstrate that such learned relationships can significantly improve the speed of a reinforcement learning algorithm in a series of Mountain Car tasks. Additionally, we demonstrate that our method may also assist with the difficult problem of task selection for transfer.

REFERENCES

• 1: A. Agnar and P. Enric. Case-based reasoning: Foundational issues, methodological variations, and system approaches, 1994.
• 2: James Sacra Albus, Brains, Behavior and Robotics, McGraw-Hill, Inc., New York, NY, 1981
• 3: Sošo Džeroski , Luc De Raedt , Kurt Driessens, Relational reinforcement learning, Machine Learning, v.43 n.1/2, p.7-52, April/May 2001 [doi>10.1023/A:1007694015589]
• 4: Fernando Fernández , Manuela Veloso, Probabilistic policy reuse in a reinforcement learning agent, Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, May 08-12, 2006, Hakodate, Japan [doi>10.1145/1160633.1160762]
• 5: M. Genesereth and N. Love. General game playing: Overview of the AAAI competition. AI Magazine, 26(2), 2005.
• 6: Gregory Kuhlmann , Peter Stone, Graph-Based Domain Mapping for Transfer Learning in General Games, Proceedings of the 18th European conference on Machine Learning, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74958-5_20]
• 7: Y. Liu and P. Stone. Value-function-based transfer for reinforcement learning using structure mapping. In Proc. of the 21st National Conf. on Artificial Intelligence, July 2006.
• 8: R. Maclin, J. Shavlik, L. Torrey, T. Walker, and E. Wild. Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In Proceedings of the 20th National Conference on Artificial Intelligence, 2005.
• 9: Dirk Ormoneit , Śaunak Sen, Kernel-Based Reinforcement Learning, Machine Learning, v.49 n.2-3, p.161-178, November-December 2002 [doi>10.1023/A:1017928328829]
• 10: M. J. D. Powell. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Computer Journal, 7:155--162, 1964.
• 11: Jan Ramon , Kurt Driessens , Tom Croonenborghs, Transfer Learning in Reinforcement Learning Problems Through Partial Policy Recycling, Proceedings of the 18th European conference on Machine Learning, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74958-5_70]
• 12: G. Rummery and M. Niranjan. On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG-RT 116, Engineering Department, Cambridge University, 1994.
• 13: M. Sharma, M. Holmes, J. C. Santamaria, A. Irani, C. Isbell, and A. Ram. Transfer learning in real-time strategy games using hybrid cbr/rl. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, 2007.
• 14: Satinder P. Singh , Richard S. Sutton, Reinforcement learning with replacing eligibility traces, Machine Learning, v.22 n.1-3, p.123-158, Jan./Feb./March 1996 [doi>10.1007/BF00114726]
• 15: V. Soni and S. Singh. Using homomorphisms to transfer options across continuous reinforcement learning domains. In Proc. of the Twenty First National Conf. on Artificial Intelligence, July 2006.
• 16: Kenneth O. Stanley , Risto Miikkulainen, Evolving neural networks through augmenting topologies, Evolutionary Computation, v.10 n.2, p.99-127, Summer 2002 [doi>10.1162/106365602320169811]
• 17: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 18: E. Talvitie and S. Singh. An experts algorithm for transfer learning. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, 2007.
• 19: Matthew E. Taylor , Peter Stone, Cross-domain transfer for reinforcement learning, Proceedings of the 24th international conference on Machine learning, p.879-886, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273607]
• 20: Matthew E. Taylor , Peter Stone , Yaxin Liu, Transfer Learning via Inter-Task Mappings for Temporal Difference Learning, The Journal of Machine Learning Research, 8, p.2125-2167, 12/1/2007
• 21: Matthew E. Taylor , Shimon Whiteson , Peter Stone, Transfer via inter-task mappings in policy search reinforcement learning, Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, May 14-18, 2007, Honolulu, Hawaii [doi>10.1145/1329125.1329170]
• 22: L. Torrey, T. Walker, J. W. Shavlik, and R. Maclin. Using advice to transfer knowledge acquired in one reinforcement learning task to another. In The 16th European Conf. on Machine Learning, 2005.
• 23: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Intelligent agents
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development

General Terms:
Algorithms, Design, Management

Keywords:
reinforcement learning, transfer learning