Fuzzy Kanerva-based function approximation for reinforcement learning

ABSTRACT

Radial Basis Functions and Kanerva Coding can give poor performance when applied to large-scale multi-agent systems. In this paper, we attempt to solve a collection of predator-prey pursuit instances and argue that the poor performance is caused by frequent prototype collisions. We show that dynamic prototype allocation and adaptation can give better results by reducing these collisions. We then describe our novel approach, fuzzy Kanerva-based function approximation, that uses a fine-grained fuzzy membership grade to describe a state-action pair's adjacency with respect to each prototype. This approach completely eliminates prototype collisions. We conclude that adaptive fuzzy Kanerva Coding can significantly improve a reinforcement learner's ability to solve large-scale multi-agent problems.

REFERENCES

• 1: James Sacra Albus, Brains, Behavior and Robotics, McGraw-Hill, Inc., New York, NY, 1981
• 2: G. Hinton. Distributed representations. Technical Report, Department of Computer Science, Carnegie-Mellon University, Pittsburgh, 1984.
• 3: Philipp W. Keller , Shie Mannor , Doina Precup, Automatic basis function construction for approximate dynamic programming and reinforcement learning, Proceedings of the 23rd international conference on Machine learning, p.449-456, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143901]
• 4: B. Ratitch and D. Precup. Sparse distributed memories for on-line value-based reinforcement learning. In Proc. of the European Conf. on Machine Learning, 2004.
• 5: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 6: Cheng Wu , Waleed M. Meleis, Adaptive Kanerva-based function approximation for multi-agent systems, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search

General Terms:
Algorithms, Experimentation

Keywords:
function approximation, fuzzy logic, reinforcement learning