Reinforcement learning for optimized trade execution

ABSTRACT

We present the first large-scale empirical application of reinforcement learning to the important problem of optimized trade execution in modern financial markets. Our experiments are based on 1.5 years of millisecond time-scale limit order data from NASDAQ, and demonstrate the promise of reinforcement learning methods to market microstructure problems. Our learning algorithm introduces and exploits a natural "low-impact" factorization of the state space.

REFERENCES

• 1: Almgren, R., Chriss, N., Optimal Execution of Portfolio Transactions, Journal of Risk, 2002.
• 2: Bertsimas, D., A, Lo, A., Optimal Control of Execution Costs. Journal of Financial Markets 1, 1--50, 1998.
• 3: Chan, N., Shelton, C., Poggio, T., An Electronic Market-Maker. Al Memo, MIT, 2001.
• 4: Coggins, R., Blazejewski, A., Aitken, M., Optimal Trade Execution of Equities in a Limit Order Market, International Conference on Computational Intelligence for Financial Engineering, pp. 371--378, March, 2003.
• 5: El-Yaniv, R., Fiat, A., Karp, R., Turin, G. Optimal Search on One-Way Trading Online Algorithms. Algorithmica 30:101--139, 2001.
• 6: Sham M. Kakade , Michael Kearns , Yishay Mansour , Luis E. Ortiz, Competitive algorithms for VWAP and limit order trading, Proceedings of the 5th ACM conference on Electronic commerce, May 17-20, 2004, New York, NY, USA [doi>10.1145/988772.988801]
• 7: Kim, A., Shelton, C., Modeling Stock Order Flows and Learning Market-Making from Data. AI Memo 2002-009, MIT, 2002.
• 8: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 9: Yuriy Nevmyvaka , Michael Kearns , Amy Papandreou , Katia Sycara, Electronic Trading in Order-Driven Markets: Efficient Execution, Proceedings of the Seventh IEEE International Conference on E-Commerce Technology (CEC'05), p.190-197, July 19-22, 2005 [doi>10.1109/ICECT.2005.42]
• 10: Gerald Tesauro , Jonathan L. Bredin, Strategic sequential bidding in auctions using dynamic programming, Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2, July 15-19, 2002, Bologna, Italy [doi>10.1145/544862.544885]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search

Additional Classification:
  J. Computer Applications
  J.1 ADMINISTRATIVE DATA PROCESSING
      Subjects: Financial (e.g., EFTS)

General Terms:
Algorithms, Theory