A social reinforcement learning agent

ABSTRACT

We report on our reinforcement learning work on Cobot, a software agent that resides in the well-known online chat community LambdaMOO. Our initial work on Cobot~\cite{cobotaaai} provided him with the ability to collect {\em social statistics\/} and report them to users in a reactive manner. Here we describe our application of reinforcement learning to allow Cobot to proactively take actions in this complex social environment, and adapt his behavior from multiple sources of human reward. After 5 months of training, Cobot received 3171 reward and punishment events from 254 different Lambda\-MOO users, and learned nontrivial preferences for a number of users. Cobot modifies his behavior based on his current state in an attempt to maximize reward. Here we describe LambdaMOO and the state and action spaces of Cobot, and report the statistical results of the learning experiment.

REFERENCES

• 1: Eisenberg, A. (2000). Find Me a File, Cache Me a Catch. New York Times, February 10, 2000. http://www.nytimes.com/library/tech/00/02/circuits/ articles/10matc.html.
• 2: Leonard N. Foner, Entertaining agents: a sociological case study, Proceedings of the first international conference on Autonomous agents, p.122-129, February 05-08, 1997, Marina del Rey, California, United States [doi>10.1145/267658.267684]
• 3: Charles Lee Isbell, Jr. , Michael J. Kearns , Dave Kormann , Satinder P. Singh , Peter Stone, Cobot in LambdaMOO: A Social Statistics Agent, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.36-41, July 30-August 03, 2000
• 4: Michael L. Mauldin, ChatterBots, TinyMuds, and the Turing test: entering the Loebner Prize competition, Proceedings of the twelfth national conference on Artificial intelligence (vol. 1), p.16-21, October 1994, Seattle, Washington, United States
• 5: Shelton, C. R. (2000). Balancing Multiple Sources of Reward in Reinforcement Learning. Submitted for publication in Neural Information Processing Systems-2000.
• 6: Satinder P. Singh , Michael J. Kearns , Diane J. Litman , Marilyn A. Walker, Empirical Evaluation of a Reinforcement Learning Spoken Dialogue System, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.645-651, July 30-August 03, 2000
• 7: Peter Stone , Manuela Veloso, Team-partitioned, opaque-transition reinforcement learning, Proceedings of the third annual conference on Autonomous Agents, p.206-212, April 1999, Seattle, Washington, United States [doi>10.1145/301136.301195]
• 8: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 9: Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems-1999.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Design, Experimentation, Measurement, Performance, Theory