Reward shaping for valuing communications during multi-agent coordination

ABSTRACT

Decentralised coordination in multi-agent systems is typically achieved using communication. However, in many cases, communication is expensive to utilise because there is limited bandwidth, it may be dangerous to communicate, or communication may simply be unavailable at times. In this context, we argue for a rational approach to communication --- if it has a cost, the agents should be able to calculate a value of communicating. By doing this, the agents can balance the need to communicate with the cost of doing so. In this research, we present a novel model of rational communication, that uses reward shaping to value communications, and employ this valuation in decentralised POMDP policy generation. In this context, reward shaping is the process by which expectations over joint actions are adjusted based on how coordinated the agent team is. An empirical evaluation of the benefits of this approach is presented in two domains. First, in the context of an idealised bench-mark problem, the multiagent Tiger problem, our method is shown to require significantly less communication (up to 30% fewer messages) and still achieves a 30% performance improvement over the current state of the art. Second, in the context of a larger-scale problem, RoboCupRescue, our method is shown to scale well, and operate without recourse to significant amounts of domain knowledge.

REFERENCES

• 1: Daniel S. Bernstein , Shlomo Zilberstein , Neil Immerman, The Complexity of Decentralized Control of Markov Decision Processes, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.32-37, June 30-July 03, 2000
• 2: Alan Carlin , Shlomo Zilberstein, Value-based observation compression for DEC-POMDPs, Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, May 12-16, 2008, Estoril, Portugal
• 3: Piotr J. Gmytrasiewicz , Edmund H. Durfee, Rational Communication in Multi-Agent Environments, Autonomous Agents and Multi-Agent Systems, v.4 n.3, p.233-272, September 2001 [doi>10.1023/A:1011495811107]
• 4: RoboCup Rescue: A Grand Challenge for Multi-Agent Systems, Proceedings of the Fourth International Conference on MultiAgent Systems (ICMAS-2000), p.5, July 10-12, 2000
• 5: S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79--86, 1951.
• 6: R. Nair, M. Tambe, M. Yokoo, D. Pynadath, and S. Marsella. Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 705--711, 2003.
• 7: Christos Papadimitriou , John N. Tsitsiklis, The complexity of Markov decision processes, Mathematics of Operations Research, v.12 n.3, p.441-450, August 1, 1987 [doi>10.1287/moor.12.3.441]
• 8: Sébastien Paquet , Ludovic Tobin , Brahim Chaib-draa, An online POMDP algorithm for complex multiagent environments, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082620]
• 9: Maayan Roth , Reid Simmons , Manuela Veloso, Reasoning about joint beliefs for execution-time communication decisions, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082593]
• 10: S. A. Williamson, E. H. Gerding, and N. R. Jennings. A principled information valuation for communications during multi-agent coordination. In Proceedings of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains, pages 137--151, 2007.
• 11: Ping Xuan , Victor Lesser, Multi-agent policies: from centralized ones to decentralized ones, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 3, July 15-19, 2002, Bologna, Italy [doi>10.1145/545056.545078]
• 12: Milind Tambe , Weixiong Zhang, Towards Flexible Teamwork in Persistent Teams: Extended Report, Autonomous Agents and Multi-Agent Systems, v.3 n.2, p.159-183, June 2000 [doi>10.1023/A:1010026728246]
• 13: Claudia V. Goldman , Shlomo Zilberstein, Optimizing information exchange in cooperative multi-agent systems, Proceedings of the second international joint conference on Autonomous agents and multiagent systems, July 14-18, 2003, Melbourne, Australia [doi>10.1145/860575.860598]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Multiagent systems

General Terms:
Algorithms, Experimentation, Theory

Keywords:
agents, communication, decentralised POMDPs