Dynamics based control with PSRs

ABSTRACT

We present an extension of the Dynamics Based Control (DBC) paradigm to environment models based on Predictive State Representations (PSRs). We show an approximate greedy version of the DBC for PSR model, EMT-PSR, and demonstrate how this algorithm can be applied to solve several control problems. We then provide some classifications and requirements of PSR environment models that are necessary for the EMT-PSR algorithm to operate.

REFERENCES

• 1: L. Chrisman. Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In Proceedings of the Tenth National Conference on Artificial Intelligence, pages 183--188, 1992.
• 2: Leslie Pack Kaelbling , Michael L. Littman , Anthony R. Cassandra, Planning and acting in partially observable stochastic domains, Artificial Intelligence, v.101 n.1-2, p.99-134, May 1998 [doi>10.1016/S0004-3702(98)00023-X]
• 3: M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In The 11th Int. Conference on Machine Learning, pages 157--163, 1994.
• 4: Michael Lederman Littman, Algorithms for sequential decision-making, Brown University, Providence, RI, 1996
• 5: M. L. Littman, T. L. Dean, and L. P. Kaelbling. On the complexity of solving Markov decision problems. In Proceedings of the 11th Annual Conference on Uncertainty in Artificial Intelligence (UAI-95), pages 394--402, 1995.
• 6: M. L. Littman, R. S. Sutton, and S. Singh. Predictive representation of state. In Advances in Neural Information Processing Systems (NIPS-14), pages 1555--1561, 2001.
• 7: Omid Madani , Steve Hanks , Anne Condon, On the undecidability of probabilistic planning and related stochastic optimization problems, Artificial Intelligence, v.147 n.1-2, p.5-34, July 2003 [doi>10.1016/S0004-3702(02)00378-8]
• 8: R. A. McCallum. Overcoming incomplete perception with utile distinction memory. In Proceedings of the 10th International Conference on Machine Learning, 1993.
• 9: K. P. Murphy. A survey of POMDP solution techniques. Technical report, University of California at Berkeley, 2000.
• 10: R. Parr and S. Russell. Approximating optimal policies for partially observable stochastic domains. In The 14th Int. Joint Conference on Artificial Intelligence, 1995.
• 11: M. L. Puterman. Markov Decision Processes. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. Wiley-Interscience, New York, 1994.
• 12: Z. Rabinovich and J. S. Rosenschein. Extended Markov Tracking with an application to control. In The Workshop on Agent Tracking: Modeling Other Agents from Observations, at AAMAS'04, pages 95--100, New-York, July 2004.
• 13: Zinovi Rabinovich , Jeffrey S. Rosenschein, Multiagent coordination by Extended Markov Tracking, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082539]
• 14: Zinovi Rabinovich , Jeffrey S. Rosenschein, On the response of EMT-based control to interacting targets and models, Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, May 08-12, 2006, Hakodate, Japan [doi>10.1145/1160633.1160718]
• 15: Zinovi Rabinovich , Jeffrey S. Rosenschein , Gal A. Kaminka, Dynamics based control with an application to area-sweeping problems, Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, May 14-18, 2007, Honolulu, Hawaii [doi>10.1145/1329125.1329274]
• 16: Satinder Singh , Michael R. James , Matthew R. Rudary, Predictive state representations: a new theory for modeling dynamical systems, Proceedings of the 20th conference on Uncertainty in artificial intelligence, p.512-519, July 07-11, 2004, Banff, Canada
• 17: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search
          Subjects: Control theory

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.11 Distributed Artificial Intelligence
            Subjects: Multiagent systems

General Terms:
Algorithms, Experimentation, Performance

Keywords:
dynamics based control, predictive state representation