Multi-policy optimization in decentralized autonomic systems

ABSTRACT

This paper addresses the challenge of multi-policy optimization in decentralized autonomic systems. We evaluate several multi-policy reinforcement learning-based optimization techniques in an urban traffic control simulation, a canonical example of a decentralized autonomic system. Our results indicate that W-learning, which learns separately for each policy and then selects between nominated actions based on current action importance, is a suitable approach for optimization towards multiple policies on non-collaborating agents in heterogeneous autonomic environments.

REFERENCES

• 1: H. Cuayáhuitl, S. Renals, O. Lemon, and H. Shimodaira. Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces. In Int. Journal of Game Theory, 2006.
• 2: J. Dowling. The Decentralised Coordination of Self-Adaptive Components for Autonomic Distributed Systems. PhD thesis, Trinity College Dublin, 2005.
• 3: M. Humphrys. Action Selection methods using Reinforcement Learning. PhD thesis, University of Cambridge, 1996.
• 4: Jeffrey O. Kephart , David M. Chess, The Vision of Autonomic Computing, Computer, v.36 n.1, p.41-50, January 2003 [doi>10.1109/MC.2003.1160055]
• 5: A. Montresor, H. Meling, and O. Baboglu. Messor: Load-balancing through a swarm of autonomous agents. In AP2PC'02.
• 6: Vinny Reynolds , Vinny Cahill , Aline Senart, Requirements for an ubiquitous computing simulation and emulation environment, Proceedings of the first international conference on Integrated internet ad hoc and sensor networks, May 30-31, 2006, Nice, France [doi>10.1145/1142680.1142682]
• 7: S. Richter. Learning traffic control - towards practical traffic control using policy gradients. Technical report, Albert-Ludwigs-Universitat Freiburg, 2006.
• 8: Gerald Tesauro, Reinforcement Learning in Autonomic Computing: A Manifesto and Case Studies, IEEE Internet Computing, v.11 n.1, p.22-30, January 2007 [doi>10.1109/MIC.2007.21]
• 9: Gerald Tesauro , David M. Chess , William E. Walsh , Rajarshi Das , Alla Segal , Ian Whalley , Jeffrey O. Kephart , Steve R. White, A Multi-Agent Systems Approach to Autonomic Computing, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, p.464-471, July 19-23, 2004, New York, New York [doi>10.1109/AAMAS.2004.23]
• 10: Marco Wiering, Multi-Agent Reinforcement Leraning for Traffic Light Control, Proceedings of the Seventeenth International Conference on Machine Learning, p.1151-1158, June 29-July 02, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence

General Terms:
Algorithms, Design, Experimentation

Keywords:
autonomic computing, decentralized systems, reinforcement learning