Learning and adaptivity in interactive recommender systems

ABSTRACT

Recommender systems are intelligent E-commerce applications that assist users in a decision-making process by offering personalized product recommendations during an interaction session. Quite recently, conversational approaches have been introduced in order to support more interactive recommendation sessions. Notwithstanding the increased interactivity offered by these approaches, the system employs an interaction strategy that is specified apriori (at design time) and followed quite rigidly during the interaction. In this paper, we present a new type of recommender system which is capable of learning autonomously an adaptive interaction strategy for assisting the users in acquiring their interaction goals. We view the recommendation process as a sequential decision problem and we model it as a Markov Decision Process (MDP). We learn a model of the user behavior, and use it to acquire the adaptive strategy using Reinforcement Learning (RL) techniques. In this context, the system learns the optimal strategy by observing the consequences of its actions on the users and also on the final outcome of the recommendation session. We apply our approach within an existing travel recommender system which uses a rigid, non-adaptive support strategy for advising a user in refining a query to a travel product catalogue. The initial results demonstrate the value of our approach and show that our system is able to improve the non-adaptive strategy in order to learn an optimal (adaptive) recommendation strategy.

REFERENCES

• 1: Gediminas Adomavicius , Alexander Tuzhilin, Personalization technologies: a process-oriented perspective, Communications of the ACM, v.48 n.10, p.83-90, October 2005 [doi>10.1145/1089107.1089109]
• 2: Gediminas Adomavicius , Alexander Tuzhilin, Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions, IEEE Transactions on Knowledge and Data Engineering, v.17 n.6, p.734-749, June 2005 [doi>10.1109/TKDE.2005.99]
• 3: David W. Aha , Len Breslow, Refining Conversational Case Libraries, Proceedings of the Second International Conference on Case-Based Reasoning Research and Development, p.267-278, July 25-27, 1997
• 4: Jan Alexandersson , Elisabeth Maier , Norbert Reithinger, A robust and efficient three-layered dialogue component for a speech-to-speech translation system, Proceedings of the seventh conference on European chapter of the Association for Computational Linguistics, March 27-31, 1995, Dublin, Ireland [doi>10.3115/976973.977000]
• 5: Liliana Ardissono , Alexander Felfernig , Gerhard Friedrich , Anna Goy , Dietmar Jannach , Giovanna Petrone , Ralph Schäfer , Markus Zanker, A framework for the development of personalized, distributed web-based configuration systems, AI Magazine, v.24 n.3, p.93-108, September 2003
• 6: J. Boger, P. Poupart, J. Hoey, C. Boutilier, G. Fernie, and A. Mihailidis. A decision-theoretic approach to task assistance for persons with dementia. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, Scotland, 2005.
• 7: Robin Burke, Hybrid Recommender Systems: Survey and Experiments, User Modeling and User-Adapted Interaction, v.12 n.4, p.331-370, November 2002 [doi>10.1023/A:1021240730564]
• 8: Nick Golovin , Erhard Rahm, Reinforcement Learning Architecture for Web Recommendations, Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'04) Volume 2, p.398, April 05-07, 2004
• 9: Nader Mirzadeh , Francesco Ricci , Mukesh Bansal, Feature Selection Methods for Conversational Recommender Systems, Proceedings of the 2005 IEEE International Conference on e-Technology, e-Commerce and e-Service (EEE'05) on e-Technology, e-Commerce and e-Service, p.772-777, March 29-April 01, 2005 [doi>10.1109/EEE.2005.75]
• 10: Vidya Narayanan , Nicholas R. Jennings, An Adaptive Bilateral Negotiation Model for E-Commerce Settings, Proceedings of the Seventh IEEE International Conference on E-Commerce Technology, p.34-41, July 19-22, 2005 [doi>10.1109/ICECT.2005.15]
• 11: C. Raju, Y. Narahari, and K. Ravikumar. Reinforcement learning applications in dynamic pricing of retail markets. cec, 0:339, 2003.
• 12: J. Reilly, K. McCarthy, L. McGinty, and B. Smyth. Incremental critiquing. Knowledge-Based Systems, 18(4-5):143--151, 2005.
• 13: Paul Resnick , Hal R. Varian, Recommender systems, Communications of the ACM, v.40 n.3, p.56-58, March 1997 [doi>10.1145/245108.245121]
• 14: F. Ricci, A. Venturini, D. Cavada, N. Mirzadeh, D. Blaas, and M. Nones. Product recommendation with interactive query management and twofold similarity. In A. Aamodt, D. Bridge, and K. Ashley, editors, ICCBR 2003, the 5th International Conference on Case-Based Reasoning, pages 479--493, Trondheim, Norway, June 23--26 2003.
• 15: Stuart J. Russell , Peter Norvig, Artificial Intelligence: A Modern Approach, Pearson Education, 2003
• 16: J. Ben Schafer , Joseph A. Konstan , John Riedl, E-Commerce Recommendation Applications, Data Mining and Knowledge Discovery, v.5 n.1-2, p.115-153, January-April 2001 [doi>10.1023/A:1009804230409]
• 17: Guy Shani , David Heckerman , Ronen I. Brafman, An MDP-Based Recommender System, The Journal of Machine Learning Research, 6, p.1265-1295, 12/1/2005
• 18: A. Stein and U. Thiel. A conversational model of multimodal interaction in information systems. In Proceedings of the 11th National Conference on Artificial Intelligence (AAAI '93), Washington DC, USA, pages 283--288. AAAI Press/MIT Press, 1993.
• 19: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 20: C. A. Thompson, M. H. Goker, and P. Langley. A personalized system for conversational recommendations. Artificial Intelligence Research, 21:393--428, 2004.
• 21: Y. Z. Wei, L. Moreau, and N. R. Jennings. Learning users' interests in a market-based recommender system. In IDEAL, pages 833--840, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.2 Types of Systems
          Subjects: Decision support (e.g., MIS)

General Terms:
Algorithms, Design, Experimentation, Verification

Keywords:
adaptivity, conversational recommender systems, markov decision process, reinforcement learning