Reinforcement learning agents with primary knowledge designed by analytic hierarchy process

ABSTRACT

This paper presents a novel model of reinforcement learning agents. A feature of our learning agent model is to integrate analytic hierarchy process (AHP) into a standard reinforcement learning agent model, which consists of three modules: state recognition, learning, and action selecting modules. In our model, AHP module is designed with primary knowledge that human intrinsically should have in order to attain a goal state. This aims at increasing promising actions of agent especially in the earlier stages of learning instead of completely random actions as in the standard reinforcement learning algorithms. We adopt profit-sharing as a reinforcement learning algorithm and demonstrate the potential of our approach on two learning problems of a pursuit problem and a Sokoban problem with deadlock in the grid-world domains, where results indicate that the learning time can be decreased considerably for the problems and our approach efficiently avoids the deadlock for the Sokoban problem. We also show that bad effect that can be usually observed by introducing a priori knowledge into reinforcement learning process can be restrained by a method that decreases a rate of using knowledge during learning.

REFERENCES

• 1: S. Arai, K. Miyazaki, and S. Kobayashi. Generating cooperative behavior by multi-agent reinforcement learning. In Proc. of the 6th European Workshop on Learning Robots, pages 111--120, 1997.
• 2: S. Arai, Katia P. Sycara, and Terry R. Payne. Experience-based reinforcement learning to acquire effective behavior in a multi-agent domain. In Proc. of the 6th Pacific Rim International Conference on Artificial Intelligence, pages 125--135, 2000.
• 3: M. Benda, V. Jagannathan, and R. Dodhiawalla. On optimal cooperation of knowledge sources. Technical Report BCS-G2010-28, Boeing Al Center, Boeing Computer Services, Bellevue, WA, 1985.
• 4: M. E. Bratman, D. Israel, and M. E. Pollack. Plans and resource-bounded practical reasoning. Computational Intelligence, 4(4):349--355, 1988.
• 5: R. A. Brooks. A robust layered control system for a mobile robot. IEEE Robotics and Automation, 2(1):14--23, 1986.
• 6: J. Culberson. Sokoban is PSPACE-complete. In Proceedings in Informatics 4, Fun With Algorithms, E. Lodi, L. Pagli and N. Santoro Eds., pages 65--76, 1999.
• 7: K. R. Dixon, R. J. Malak, and P. K. Khosla. Incorporating prior knowledge and previously learned information into reinforcement learning. Technical Report, Institute for Complex Enginerred Systems, Carnegie Mellon University, 2000.
• 8: J.J. Grefenstette, Credit assignment in rule discovery systems based on genetic algorithms, Machine Language, v.3 n.2-3, p.225-245, Oct. 1988 [doi>10.1007/BF00113898]
• 9: L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: a survey. Journal of Artificial Intelligence Research, 4:237--285, 1996.
• 10: K. Miyazaki, M. Yamamura, and S. Kobayashi. On the rationality of profit sharing in reinforcement learning. In Proc. of the 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing, pages 285--288, 1994.
• 11: Norihiko Ono , Kenji Fukumoto, A Modular Approach to Multi-Agent Reinforcement Learning, Selected papers from the Workshop on Distributed Artificial Intelligence Meets Machine Learning, Learning in Multi-Agent Environments, p.25-39, August 13, 1996
• 12: T. Saaty. The analytic hierarchy process. The McGraw-Hill Companies, 1980.
• 13: Sandip Sen , Mahendra Sekaran, Multiagent Coordination with Learning Classifier Systems, Proceedings of the Workshop on Adaption and Learning in Multi-Agent Systems, p.218-234, August 21, 1995
• 14: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 15: M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proc. of 10th International Conference on Machine Learning, pages 330--337, 1993.
• 16: T. Unemi. Scaling up reinforcement learning with human knowledge as an intrinsic behavior. In Proc. of the 6th International Conference on Intelligent Autonomous Systems, pages 511--518, 2000.
• 17: Christopher J. C. H. Watkins , Peter Dayan, Technical Note: $\backslash cal\; Q$-Learning, Machine Learning, v.8 n.3-4, p.279-292, May 1992 [doi>10.1007/BF00992698]
• 18: Steven D. Whitehead , Dana H. Ballard, Active perception and reinforcement learning, Neural Computation, v.2 n.4, p.409-419, Winter 1990 [doi>10.1162/neco.1990.2.4.409]

INDEX TERMS

Keywords:
analytic hierarchy process, deadlock, profit sharing, pursuit problem, reinforcement learning, sokoban