Online kernel selection for Bayesian reinforcement learning

ABSTRACT

Kernel-based Bayesian methods for Reinforcement Learning (RL) such as Gaussian Process Temporal Difference (GPTD) are particularly promising because they rigorously treat uncertainty in the value function and make it easy to specify prior knowledge. However, the choice of prior distribution significantly affects the empirical performance of the learning agent, and little work has been done extending existing methods for prior model selection to the online setting. This paper develops Replacing-Kernel RL, an online model selection method for GPTD using sequential Monte-Carlo methods. Replacing-Kernel RL is compared to standard GPTD and tile-coding on several RL domains, and is shown to yield significantly better asymptotic performance for many different kernel families. Furthermore, the resulting kernels capture an intuitively useful notion of prior state covariance that may nevertheless be difficult to capture manually.

REFERENCES

• 1: Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 2: Engel, Y. (2005). Algorithms and representations for reinforcement learning. Doctoral dissertation, Hebrew University.
• 3: Yaakov Engel , Shie Mannor , Ron Meir, Reinforcement learning with Gaussian processes, Proceedings of the 22nd international conference on Machine learning, p.201-208, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102377]
• 4: Marc G. Genton, Classes of kernels for machine learning: a statistics perspective, The Journal of Machine Learning Research, 2, 3/1/2002
• 5: Gordon, N. J., Salmond, D. J., & Smith, A. F. M. (1993). Novel approach to nonlinear/non-gaussian bayesian state estimation. Radar and Signal Processing, IEE Proceedings F, 140, 107--113.
• 6: Hastie, T., Tibshirani, R., & Friedman, J. H. (2001). The elements of statistical learning. Springer.
• 7: Jung, T., & Polani, D. (2006). Least squares svm for least squares td learning. ECAI (pp. 499--503). IOS Press.
• 8: Leffler, B. R., Littman, M. L., & Edmunds, T. (2007). Efficient reinforcement learning with relocatable action models. Proc. of AAAI-07 (pp. 572--577). Menlo Park, CA, USA: The AAAI Press.
• 9: Loth, M., Davy, M., & Preux, P. (2007). Sparse temporal difference learning using lasso. IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning. Hawaii, USA.
• 10: Carl Edward Rasmussen , Christopher K. I. Williams, Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2005
• 11: Seeger, M. (2001). Covariance kernels from bayesian generative models. NIPS (pp. 905--912). MIT Press.
• 12: Alex J. Smola , Bernhard Schölkopf, A tutorial on support vector regression, Statistics and Computing, v.14 n.3, p.199-222, August 2004 [doi>10.1023/B:STCO.0000035301.49549.88]
• 13: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 14: White, A. (2007). The University of Alberta Reinforcement Learning Library. http://rlai.cs.ualberta.ca/RLR/. Edmonton, Alberta: University of Alberta.
• 15: Shimon Whiteson , Peter Stone, Evolutionary Function Approximation for Reinforcement Learning, The Journal of Machine Learning Research, 7, p.877-917, 12/1/2006

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.2 Modes of Computation
          Subjects: Online computation

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo)

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis

General Terms:
Algorithms, Design, Performance, Theory