Neuroevolutionary reinforcement learning for generalized helicopter control

ABSTRACT

Helicopter hovering is an important challenge problem in the field of reinforcement learning. This paper considers several neuroevolutionary approaches to discovering robust controllers for a generalized version of the problem used in the 2008 Reinforcement Learning Competition, in which wind in the helicopter's environment varies from run to run. We present the simple model-free strategy that won first place in the competition and also describe several more complex model-based approaches. Our empirical results demonstrate that neuroevolution is effective at optimizing the weights of multi-layer perceptrons, that linear regression is faster and more effective than evolution for learning models, and that model-based approaches can outperform the simple model-free strategy, especially if prior knowledge is used to aid model learning.

REFERENCES

• 1: P. Abbeel, V. Ganapathi, and A. Y. Ng. Learning vehicular dynamics with application to modeling helicopters. In Proceedings of Neural Information Processing Systems (NIPS), 2006.
• 2: Pieter Abbeel , Andrew Y. Ng, Exploration and apprenticeship learning in reinforcement learning, Proceedings of the 22nd international conference on Machine learning, p.1-8, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102352]
• 3: J. Bagnell and J. Schneider. Autonomous helicopter control using reinforcement learning policy search methods. In Proceedings of the International Conference on Robotics and Automation 2001. IEEE, May 2001.
• 4: T. Beielstein and S. Markon. Threshold selection, hypothesis tests and DOE methods. In 2002 Congresss on Evolutionary Computation, pages 777--782, 2002.
• 5: Richard Ernest Bellman, Dynamic Programming, Dover Publications, Incorporated, 2003
• 6: David E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 7: David E. Goldberg , Jon Richardson, Genetic algorithms with sharing for multimodal function optimization, Proceedings of the Second International Conference on Genetic Algorithms on Genetic algorithms and their application, p.41-49, October 1987, Cambridge, Massachusetts, United States
• 8: F. Gomez, J. Schmidhuber, and R. Miikkulainen. Efficient non-linear control through neuroevolution. In Proceedings of the European Conference on Machine Learning, 2006.
• 9: Frédéric Gruau , Darrell Whitley , Larry Pyeatt, A comparison between cellular encoding and direct encoding for genetic neural networks, Proceedings of the First Annual Conference on Genetic Programming, July 28-31, 1996, Stanford, California
• 10: L. P. Kaelbling, M. L. Littman, and A. P. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237--285, 1996.
• 11: R. Koppejan. Neuroevolutionary reinforcement learning for generalized helicopter control. Master's thesis, Universiteit van Amsterdam, 2009.
• 12: Andrew W. Moore , Christopher G. Atkeson, Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time, Machine Learning, v.13 n.1, p.103-130, Oct. 1993 [doi>10.1023/A:1022635613229]
• 13: D. E. Moriarty, A. C. Schultz, and J. J. Grefenstette. Evolutionary algorithms for reinforcement learning. Journal of Artificial Intel ligence Research, 11:199--229, 1999.
• 14: A. Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang. Inverted autonomous helicopter flight via reinforcement learning. In Proceedings of the International Symposium on Experimental Robotics, 2004.
• 15: A. Y. Ng, H. J. Kim, M. I. Jordan, and S. Sastry. Autonomous helicopter flight via reinforcement learning. In Advances in Neural Information Processing Systems 16, 2004.
• 16: Peter Stagge, Averaging Efficiently in the Presence of Noise, Proceedings of the 5th International Conference on Parallel Problem Solving from Nature, p.188-200, September 27-30, 1998
• 17: Kenneth O. Stanley , Risto Miikkulainen, Evolving neural networks through augmenting topologies, Evolutionary Computation, v.10 n.2, p.99-127, Summer 2002 [doi>10.1162/106365602320169811]
• 18: P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning in robocup-soccer keepaway. Adaptive Behavior, 13(3):165--188, 2005.
• 19: Richard S. Sutton, Learning to Predict by the Methods of Temporal Differences, Machine Learning, v.3 n.1, p.9-44, August 1988 [doi>10.1023/A:1022633531479]
• 20: Richard S. Sutton, Integrated architecture for learning, planning, and reacting based on approximating dynamic programming, Proceedings of the seventh international conference (1990) on Machine learning, p.216-224, June 1990, Austin, Texas, United States
• 21: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 22: Gerald Tesauro, Temporal difference learning and TD-Gammon, Communications of the ACM, v.38 n.3, p.58-68, March 1995 [doi>10.1145/203330.203343]
• 23: Shimon Whiteson , Nate Kohl , Risto Miikkulainen , Peter Stone, Evolving Soccer Keepaway Players Through Task Decomposition, Machine Learning, v.59 n.1-2, p.5-30, May 2005 [doi>10.1007/s10994-005-0460-9]
• 24: Shimon Whiteson , Peter Stone, Evolutionary Function Approximation for Reinforcement Learning, The Journal of Machine Learning Research, 7, p.877-917, 12/1/2006
• 25: X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423--1447, 1999.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.9 Robotics

General Terms:
Algorithms, Experimentation

Keywords:
evolutionary computation, neural networks, reinforcement learning, robot control