Monte-Carlo simulation balancing

ABSTRACT

In this paper we introduce the first algorithms for efficiently learning a simulation policy for Monte-Carlo search. Our main idea is to optimise the balance of a simulation policy, so that an accurate spread of simulation outcomes is maintained, rather than optimising the direct strength of the simulation policy. We develop two algorithms for balancing a simulation policy by gradient descent. The first algorithm optimises the balance of complete simulations, using a policy gradient algorithm; whereas the second algorithm optimises the balance over every two steps of simulation. We compare our algorithms to reinforcement learning and supervised learning algorithms for maximising the strength of the simulation policy. We test each algorithm in the domain of 5 x 5 and 6 x 6 Computer Go, using a softmax policy that is parameterised by weights for a hundred simple patterns. When used in a simple Monte-Carlo search, the policies learnt by simulation balancing achieved significantly better performance, with half the mean squared error of a uniform random policy, and similar overall performance to a sophisticated Go engine.

REFERENCES

• 1: Darse Billings , Lourdes Peña , Jonathan Schaeffer , Duane Szafron, Using probabilistic knowledge and simulation to play poker, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.697-703, July 18-22, 1999, Orlando, Florida, United States
• 2: Chaslot, G., Winands, M., Szita, I., & van den Herik, H. (2008). Parameter tuning by the cross-entropy method. 8th European Workshop on Reinforcement Learning.
• 3: Coulom, R. (2007). Computing Elo ratings of move patterns in the game of Go. Computer Games Workshop.
• 4: Finnsson, H., & Bjöörnsson, Y. (2008). Simulation-based approach to general game playing. 23rd Conference on Artificial Intelligence (pp. 259--264).
• 5: Sylvain Gelly , David Silver, Combining online and offline knowledge in UCT, Proceedings of the 24th international conference on Machine learning, p.273-280, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273531]
• 6: Gelly, S., Wang, Y., Munos, R., & Teytaud, O. (2006). Modification of UCT with patterns in Monte-Carlo Go (Technical Report 6062). INRIA.
• 7: Kocsis, L., & Szepesvari, C. (2006). Bandit based Monte-Carlo planning. 15th European Conference on Machine Learning (pp. 282--293).
• 8: Brian Sheppard, World-championship-caliber Scrabble, Artificial Intelligence, v.134 n.1-2, p.241-275, January 2002 [doi>10.1016/S0004-3702(01)00166-7]
• 9: Silver, D., Sutton, R., & Müüller, M. (2007). Reinforcement learning of local shape in the game of Go. 20th International Joint Conference on Artificial Intelligence (pp. 1053--1058).
• 10: Tesauro, G., & Galperin, G. (1996). On-line policy improvement using Monte-Carlo search. Advances in Neural Information Processing 9 (pp. 1068--1074).
• 11: Ronald J. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning, v.8 n.3-4, p.229-256, May 1992 [doi>10.1007/BF00992696]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.8 Types of Simulation
          Subjects: Monte Carlo

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.6 Optimization
          Subjects: Gradient methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development
          Subjects: Modeling methodologies
      I.6.6 Simulation Output Analysis

General Terms:
Algorithms, Design, Performance, Theory