Learning complex motions by sequencing simpler motion templates

ABSTRACT

Abstraction of complex, longer motor tasks into simpler elemental movements enables humans and animals to exhibit motor skills which have not yet been matched by robots. Humans intuitively decompose complex motions into smaller, simpler segments. For example when describing simple movements like drawing a triangle with a pen, we can easily name the basic steps of this movement.

Surprisingly, such abstractions have rarely been used in artificial motor skill learning algorithms. These algorithms typically choose a new action (such as a torque or a force) at a very fast time-scale. As a result, both policy and temporal credit assignment problem become unnecessarily complex - often beyond the reach of current machine learning methods.

We introduce a new framework for temporal abstractions in reinforcement learning (RL), i.e. RL with motion templates. We present a new algorithm for this framework which can learn high-quality policies by making only few abstract decisions.

REFERENCES

• 1: Arbib, M. A. (1981). Perceptual structures and distributed motor control. Handbook of physiology, section 2: The nervous system vol. ii, motor control, part 1, 1449--1480.
• 2: Atkeson, C., & Stephens, B. (2007). Multiple balance strategies from one optimization criterion. 7th IEEE-RAS International Conference on Humanoid Robots.
• 3: Christopher G. Atkeson , Andrew W. Moore , Stefan Schaal, Locally Weighted Learning, Artificial Intelligence Review, v.11 n.1-5, p.11-73, Feb. 1997 [doi>10.1023/A:1006559212014]
• 4: Bradtke, S. J., & Duff, M. O. (1995). Reinforcement learning methods for continuous-time markov decision problems. Advances in Neural Information Processing Systems 7, 7, 393--400.
• 5: Kerstin Dautenhahn , Chrystopher L. Nehaniv, Imitation in animals and artifacts, MIT Press, Cambridge, MA, 2002
• 6: Damien Ernst , Pierre Geurts , Louis Wehenkel, Tree-Based Batch Mode Reinforcement Learning, The Journal of Machine Learning Research, 6, p.503-556, 12/1/2005
• 7: Ghavamzadeh, M., & Mahadevan, S. (2003). Hierarchical policy gradient algorithms. Twentieth International Conference on Machine Learning (ICML-2003) (pp. 226--233).
• 8: Huber, M., & Grupen, R. A. (1998). Learning robot control---using control policies as abstract actions. In NIPS'98 Workshop: Abstraction and Hierarchy in Reinforcement Learning.
• 9: Ijspeert, A., Nakanishi, J., & Schaal, S. (2002). Learning attractor landscapes for learning motor primitives. Advances in Neural Information Processing Systems 15 (NIPS2002) (pp. 1523--1530).
• 10: Kober, J., & Peters, J. (2009). Policy search for motor primitives in robotics. Advances in Neural Information Processing Systems 22 (NIPS 2008) (pp. 849--856). MA: MIT Press.
• 11: Neumann, G., & Peters, J. (2009). Fitted Q-iteration by Advantage Weighted Regression. Advances in Neural Information Processing Systems 22 (NIPS 2008) (pp. 1177--1184). MA: MIT Press.
• 12: Riedmiller, M. (2005). Neural fitted Q-iteration - first experiences with a data efficient neural reinforcement learning method. Proceedings of the European Conference on Machine Learning (ECML) (pp. 317--328).
• 13: Richard S. Sutton , Doina Precup , Satinder Singh, Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning, Artificial Intelligence, v.112 n.1-2, p.181-211, Aug. 1999 [doi>10.1016/S0004-3702(99)00052-1]
• 14: Xu, X., & Antsaklis, P. (2002). An approach to optimal control of switched systems with internally forced switchings. Proceedings of the American Control Conference (pp. 148--153). Anchorage, USA.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory