Fuzzy CMAC with automatic state partition for reinforcementlearning

ABSTRACT

Most of reinforcement learning (RL) algorithms use value function to seek the optimal policy. In large or even continuous states, function approximation approaches must be used to represent value function. The structures of function approximators influence the learning performance greatly. However, the design of structures relies too much on human designer and inappropriate design can lead to poor performance. In this paper, we propose a novel function approximator called Fuzzy CMAC (FCMAC) with automatic state partition (ASP-FCMAC) to automate the structure design for FCMAC. Based on CMAC (also known as tile coding), ASP-FCMAC employs fuzzy membership function to lower the computation load, and analyzes Bellman error as well as learning weights to partition the state automatically so as to generate the structure of FCMAC. Empirical results in both mountain car and RoboCup Keepaway domains demonstrate that ASP-FCMAC can automatically generate the structure of FCMAC and agent using it can learn efficiently.

REFERENCES

• 1: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 2: J. C Santamaria , Richard S. Sutton , Ashwin Ram, Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces, University of Massachusetts, Amherst, MA, 1996
• 3: Peter Stone, Richard S. Sutton, and Gregory Kuhlmann. Reinforcement learning for RoboCup-soccer keepaway. Adaptive Behavior, 13(3):165--188, 2005.
• 4: Peter Stone, Gregory Kuhlmann, Matthew E. Taylor, and Yaxin Liu. Keepaway soccer: From machine learning testbed to benchmark. In Itsuki Noda, Adam Jacoff, Ansgar Bredenfeld, and Yasutake Takahashi, editors, RoboCup-2005: Robot Soccer World Cup IX, volume 4020, pages 93--105, Berlin, 2006. Springer Verlag.
• 5: Shimon Whiteson , Peter Stone, Evolutionary Function Approximation for Reinforcement Learning, The Journal of Machine Learning Research, 7, p.877-917, 12/1/2006
• 6: Philipp W. Keller , Shie Mannor , Doina Precup, Automatic basis function construction for approximate dynamic programming and reinforcement learning, Proceedings of the 23rd international conference on Machine learning, p.449-456, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143901]
• 7: L. Tokarchuk. Fuzzy and tile coding approximation techniques for coevolution in reinforcement learning. Technical report, University of London, 2006.
• 8: Shimon Whiteson, Matthew E. Taylor, and Peter Stone. Adaptive tile coding for value function approximation. Technical Report AI-TR-07-339, University of Texas at Austin, 2007.
• 9: Alexander A. Sherstov and Peter Stone. Function approximation via tile coding: Automating parameter choice. In J.-D. Zucker and I. Saitta, editors, SARA 2005, volume 3607 of Lecture Notes in Artificial Intelligence, pages 194--205, Berlin, 2005. Springer Verlag.
• 10: Jan Hendrik Metzen , Mark Edgington , Yohannes Kassahun , Frank Kirchner, Analysis of an evolutionary reinforcement learning method in a multiagent domain, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal
• 11: Junhong Nie , D. A. Linkens, FCMAC: a fuzzified cerebellar model articulation controller with self-organizing capacity, Automatica (Journal of IFAC), v.30 n.4, p.655-664, April 1994 [doi>10.1016/0005-1098(94)90154-6]
• 12: S.F. Su, Z.J. Lee, and Y.P. Wang. Robust and fast learning for fuzzy cerebellar model articulation controllers. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 36(1):203 -- 208, 2006.
• 13: Pier Luca Lanzi , Daniele Loiacono , Stewart W. Wilson , David E. Goldberg, Classifier prediction based on tile coding, Proceedings of the 8th annual conference on Genetic and evolutionary computation, July 08-12, 2006, Seattle, Washington, USA [doi>10.1145/1143997.1144242]
• 14: Sridhar Mahadevan. Samuel meets amarel: Automating value function approximation using global state space analysis. In Proceedings of AAAI, pages 1000--1005, 2005.
• 15: Ishai Menache, Shie Mannor, and Nahum Shimkin. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research, 134:215--238, 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search
          Subjects: Heuristic methods

General Terms:
Algorithms

Keywords:
automatic state partition, fuzzy CMAC, reinforcement learning