Towards continuous actions in continuous space and time using self-adaptive constructivism in neural XCSF

ABSTRACT

This paper presents a Learning Classifier System (LCS) where each classifier condition is represented by a feed-forward multi-layered perceptron (MLP) network. Adaptive behavior is realized through the use of self-adaptive parameters and neural constructivism, providing the system with a flexible knowledge representation. The approach allows for the evolution of networks of appropriate complexity to solve a continuous maze environment, here using either discrete-valued actions, continuous-valued actions, or continuous-valued actions of continuous duration. In each case, it is shown that the neural LCS employed is capable of developing optimal solutions to the reinforcement learning task presented in this paper.

REFERENCES

• 1: Ahluwalia, M.&Bull, L. 1999. A Genetic Programming Classifier System. In W. Banzhaf, J. Daida, A.E. Eiben, M.H. Garzon, V. Honavar, M. Jakiela&R.E. Smith (Eds.) Proceedings of the Genetic and Evolutionary Computation Conference GECCO--99. San Mateo, CA: Morgan Kaufmann, pp11--18.
• 2: Belue, L. M.&Bauer Jr., K. W. 1995. Determining input features for multilayer perceptrons. Neurocomputing 7:111--121.
• 3: Andrea Bonarini , Claudio Bonacina , Matteo Matteucci, Fuzzy and Crisp Representations of Real-Valued Input for Learning Classifier Systems, Learning Classifier Systems, From Foundations to Applications, p.107-124, January 2000
• 4: Boyan, J. A.&Moore, A. W. 1995. Generalization in reinforcement learning: Safely approximating the value function. In G. Tesauro, D. S. Touretzky, and T. K. Leen, editors, Advances in Neural Information Processing Systems 7, the MIT Press, Cambridge, MA , pp369--376.
• 5: Larry Bull, On Using Constructivism in Neural Classifier Systems, Proceedings of the 7th International Conference on Parallel Problem Solving from Nature, p.558-567, September 07-11, 2002
• 6: Larry Bull , Toby O'Hara, Accuracy-based Neuro And Neuro-fuzzy Classifier Systems, Proceedings of the Genetic and Evolutionary Computation Conference, p.905-911, July 09-13, 2002
• 7: Bull, L., Hurst, J.,&Tomlinson, A. 2000. Self-Adaptive Mutation in Classifier System Controllers. In J-A. Meyer, A. Berthoz, D. Floreano, H. Roitblatt&S.W. Wilson (Eds.) From Animals to Animats 6 -- The Sixth International Conference on the Simulation of Adaptive Behaviour, MIT Press.
• 8: Martin V. Butz , Oliver Herbort, Context-dependent predictions and cognitive arm control with XCSF, Proceedings of the 10th annual conference on Genetic and evolutionary computation, July 12-16, 2008, Atlanta, GA, USA [doi>10.1145/1389095.1389360]
• 9: Carse, B.&Fogarty, T.C. 1994. A Delayed-Action Classifier System for Learning in Temporal Environments. International Conference on Evolutionary Computation: 670--673.
• 10: Casillas, J. Carse, B.&Bull, L. 2007. Fuzzy XCS: A Michigan Genetic Fuzzy System. IEEE Transactions on Fuzzy Systems 15(4): 536--550.
• 11: Inman Harvey , Phil Husbands , Dave Cliff, Seeing the light: artificial evolution, real vision, Proceedings of the third international conference on Simulation of adaptive behavior : from animals to animats 3: from animals to animats 3, p.392-401, July 1994, Brighton, United Kingdom
• 12: John H. Holland, Adaptation in natural and artificial systems, MIT Press, Cambridge, MA, 1992
• 13: Holland, J. H. 1976. Adaptation. In R. Rosen&F. M. Snell (Eds.) Progress in Theoretical Biology 4. New York: Academic Press, pp263--293.
• 14: Gerard David Howard , Larry Bull, On the effects of node duplication and connection-oriented constructivism in neural XCSF, Proceedings of the 2008 GECCO conference companion on Genetic and evolutionary computation, July 12-16, 2008, Atlanta, GA, USA [doi>10.1145/1388969.1389010]
• 15: Gerard D. Howard , Larry Bull , Pier-Luca Lanzi, Self-adaptive constructivism in Neural XCS and XCSF, Proceedings of the 10th annual conference on Genetic and evolutionary computation, July 12-16, 2008, Atlanta, GA, USA [doi>10.1145/1389095.1389364]
• 16: Jacob Hurst , Larry Bull, A Neural Learning Classifier System with Self-Adaptive Constructivism for Mobile Robot Control, Artificial Life, v.12 n.3, p.353-380, July 2006 [doi>10.1162/artl.2006.12.3.353]
• 17: Jacob Hurst , Larry Bull , Chris Melhuish, TCS Learning Classifier System Controller on a Real Robot, Proceedings of the 7th International Conference on Parallel Problem Solving from Nature, p.588-600, September 07-11, 2002
• 18: Ron Kohavi , George H. John, Wrappers for feature subset selection, Artificial Intelligence, v.97 n.1-2, p.273-324, Dec. 1997 [doi>10.1016/S0004-3702(97)00043-X]
• 19: Pier Luca Lanzi, An analysis of generalization in the xcs classifier system, Evolutionary Computation, v.7 n.2, p.125-149, Summer 1999 [doi>10.1162/evco.1999.7.2.125]
• 20: Lanzi, P. L., Loiacono, D., Wilson, S. W.&Goldberg, D. E. 2005. XCS with computed prediction in continuous multistep environments. In Proceedings of the IEEE Congress on Evolutionary Computation CEC-2005, IEEE, Edinburgh, UK, pp2032--2039.
• 21: Quartz, S. R.&Sejnowski, T. J. 1997. The Neural Basis of Cognitive Development: A Constructionist Manifesto. Behavioural and Brain Sciences 20(4): 537--596.
• 22: David E. Rumelhart , James L. McClelland , CORPORATE PDP Research Group, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 23: Schlessinger, E., Bentley, P. J., Lotto, R. B. 2005. Analysing the Evolvability of Neural Network Agents through Structural Mutations. Proc. of European Conference on Artificial Life (ECAL 2005), September 5--9, 2005, Canterbury, UK.
• 24: Studley, M.&Bull, L. 2005. X-TCS: Accuracy-based Learning Classifier System Robotics. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE, pp2099--2106.
• 25: Andy Tomlinson , Larry Bull, A Corporate XCS, Learning Classifier Systems, From Foundations to Applications, p.195-208, January 2000
• 26: Hau Trung Tran , Cédric Sanza , Yves Duthen , Thuc Dinh Nguyen, XCSF with computed continuous action, Proceedings of the 9th annual conference on Genetic and evolutionary computation, July 07-11, 2007, London, England [doi>10.1145/1276958.1277327]
• 27: Valenzuela-Rendón, M. 1991. The Fuzzy Classifier System: a Classifier System for Continuously Varying Variables. In Proceedings of the 4th International Conference on Genetic Algorithms (ICGA91) pp346--353.
• 28: Shimon Whiteson , Peter Stone , Kenneth O. Stanley , Risto Miikkulainen , Nate Kohl, Automatic feature selection in neuroevolution, Proceedings of the 2005 conference on Genetic and evolutionary computation, June 25-29, 2005, Washington DC, USA [doi>10.1145/1068009.1068210]
• 29: Stewart W. Wilson, Zcs: A zeroth level classifier system, Evolutionary Computation, v.2 n.1, p.1-18, Spring 1994 [doi>10.1162/evco.1994.2.1.1]
• 30: Stewart W. Wilson, Classifier fitness based on accuracy, Evolutionary Computation, v.3 n.2, p.149-175, Summer 1995 [doi>10.1162/evco.1995.3.2.149]
• 31: Wilson, S. W. 2001. Function Approximation with a Classifier System. In Spector, L., D., G. E., Wu, A., Langdon, W.B., Voight, H. M., and Gen, M., (Eds.) Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 01) Morgan Kaufmann. pp974--981.
• 32: Wilson, S. W. 2007. Three Architectures for Continuous Action. Learning Classifier Systems. International Workshops, IWLCS 2003-2005, Revised Selected Papers. In T. Kovacs, X. Llorà, K. Takadama, P. L. Lanzi, W. Stolzmann, S. W. Wilson (Eds.) Lecture Notes in Artificial Intelligence (LNAI-4399),. Berlin, Springer-Verlag. pp. 239--257.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Knowledge acquisition

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Connectionism and neural nets; Parameter learning

General Terms:
Experimentation

Keywords:
constructivism, continuous environments, learning classifier systems, neural networks, self-adaptation