Transfer via soft homomorphisms

ABSTRACT

The field of transfer learning aims to speed up learning across multiple related tasks by transferring knowledge between source and target tasks. Past work has shown that when the tasks are specified as Markov Decision Processes (MDPs), a function that maps states in the target task to similar states in the source task can be used to transfer many types of knowledge. Current approaches for autonomously learning such functions are inefficient or require domain knowledge and lack theoretical guarantees of performance. We devise a novel approach that learns a stochastic mapping between tasks. Using this mapping, we present two algorithms for autonomous transfer learning -- one that has strong convergence guarantees and another approximate method that learns online from experience. Extending existing work on MDP homomorphisms, we present theoretical guarantees for the quality of a transferred value function.

REFERENCES

• 1: Robert Givan , Thomas Dean , Matthew Greig, Equivalence notions and model minimization in Markov decision processes, Artificial Intelligence, v.147 n.1-2, p.163-223, July 2003 [doi>10.1016/S0004-3702(02)00376-4]
• 2: G. Konidaris and A. G. Barto. Building portable options: Skill transfer in reinforcement learning. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, pages 895--900, 2007.
• 3: Yaxin Liu , Peter Stone, Value-function-based transfer for reinforcement learning using structure mapping, Proceedings of the 21st national conference on Artificial intelligence, p.415-420, July 16-20, 2006, Boston, Massachusetts
• 4: Luís F. Portugal , Joaquim J. Júdice , Luís N. Vicente, A comparison of block pivoting and interior-point algorithms for linear least squares problems with nonnegative variables, Mathematics of Computation, v.63 n.208, p.625-643, Oct. 1994 [doi>10.2307/2153286]
• 5: Balaraman Ravindran , Andrew G. Barto, An algebraic approach to abstraction in reinforcement learning, University of Massachusetts Amherst, 2004
• 6: Satinder P. Singh , Richard C. Yee, An Upper Bound on the Loss from Approximate Optimal-Value Functions, Machine Learning, v.16 n.3, p.227-233, Sept. 1994 [doi>10.1023/A:1022693225949]
• 7: S. P. Singh, T. Jaakkola, and M. I. Jordan. Reinforcement learning with soft state aggregation. In Advances in Neural Information Processing Systems, volume 7, pages 361--368, 1995.
• 8: Vishal Soni , Satinder Singh, Using Homomorphisms to transfer options across continuous reinforcement learning domains, Proceedings of the 21st national conference on Artificial intelligence, p.494-499, July 16-20, 2006, Boston, Massachusetts
• 9: Matthew E. Taylor , Gregory Kuhlmann , Peter Stone, Autonomous transfer for reinforcement learning, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal
• 10: M. E. Taylor, P. Stone, and Y. Liu. Value functions for RL-based behavior transfer: A comparative study. In Proceedings of the Twentieth National Conference on Artificial Intelligence, pages 880--885, July 2005.
• 11: Matthew E. Taylor , Shimon Whiteson , Peter Stone, Transfer via inter-task mappings in policy search reinforcement learning, Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, May 14-18, 2007, Honolulu, Hawaii [doi>10.1145/1329125.1329170]
• 12: Alicia Peregrin Wolfe , Andrew G. Barto, Decision tree methods for finding reusable MDP homomorphisms, Proceedings of the 21st national conference on Artificial intelligence, p.530-535, July 16-20, 2006, Boston, Massachusetts

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Algorithms

Keywords:
Markov decision process, homomorphism, transfer learning