Adapting the interaction state model in conversational recommender systems

ABSTRACT

Conventional conversational recommender systems support interaction strategies that are hard-coded into the system in advance. In this context, Reinforcement Learning techniques have been proposed to learn an optimal, user-adapted interaction strategy, by encoding relevant information as features describing the state of the interaction. In this regard, a crucial problem is to select this subset of relevant features from a larger set, for any given recommendation task. In this paper, we tackle this issue of state features selection by proposing and exploiting two criteria for determining feature relevancy. Our results show that adding a feature might not always be beneficial, that the relevancy is influenced by the user behavior, and also by the numerical reinforcement signal which is exploited by the adaptive system for learning the optimal strategy. These results, obtained in off-line simulations and in a simplified scenario, were exploited to design an adaptive recommender system for an online travel planning application.

REFERENCES

• 1: Gediminas Adomavicius , Alexander Tuzhilin, Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions, IEEE Transactions on Knowledge and Data Engineering, v.17 n.6, p.734-749, June 2005 [doi>10.1109/TKDE.2005.99]
• 2: Derek Bridge , Mehmet H. Göker , Lorraine McGinty , Barry Smyth, Case-based recommender systems, The Knowledge Engineering Review, v.20 n.3, p.315-320, September 2005 [doi>10.1017/S0269888906000567]
• 3: Matthew Frampton , Oliver Lemon, Learning more effective dialogue strategies using limited dialogue move features, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, p.185-192, July 17-18, 2006, Sydney, Australia [doi>10.3115/1220175.1220199]
• 4: Nick Golovin , Erhard Rahm, Reinforcement Learning Architecture for Web Recommendations, Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC'04) Volume 2, p.398, April 05-07, 2004
• 5: Ron Kohavi , George H. John, Wrappers for feature subset selection, Artificial Intelligence, v.97 n.1-2, p.273-324, Dec. 1997 [doi>10.1016/S0004-3702(97)00043-X]
• 6: Tariq Mahmood , Francesco Ricci, Learning and adaptivity in interactive recommender systems, Proceedings of the ninth international conference on Electronic commerce, August 19-22, 2007, Minneapolis, MN, USA [doi>10.1145/1282100.1282114]
• 7: T. Mahmood and F. Ricci. Towards learning user-adaptive state models in a conversational recommender system. In ABIS'07: Proceedings 15th Workshop on Adaptivity and User Modeling in Interactive Systems, Halle, Germany, 2007.
• 8: T. Mahmood, F. Ricci, A. Venturini, and W. Höpken. Adaptive recommender systems for travel planning. In W. H. Peter OConnor and U. Gretzel, editors, Information and Communication Technologies in Tourism 2008, proceedings of ENTER 2008 International Conference, pages 1--11, Innsbruck, 2008. Springer.
• 9: Nader Mirzadeh , Francesco Ricci, COOPERATIVE QUERY REWRITING FOR DECISION MAKING SUPPORT AND RECOMMENDER SYSTEMS, Applied Artificial Intelligence, v.21 n.10, p.895-932, November 2007 [doi>10.1080/08839510701527515]
• 10: Guy Shani , David Heckerman , Ronen I. Brafman, An MDP-Based Recommender System, The Journal of Machine Learning Research, 6, p.1265-1295, 12/1/2005
• 11: H. Shimazu. ExpertClerk: Navigating shoppers buying process with the combination of asking and proposing. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, Washington, USA, 2001.
• 12: S. P. Singh, M. J. Kearns, D. J. Litman, and M. A. Walker. Reinforcement learning for spoken dialogue systems. In NIPS Conference, Colorado, USA, 1999.
• 13: Richard S. Sutton , Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, 1998
• 14: Nima Taghipour , Ahmad Kardan, A hybrid web recommender system based on Q-learning, Proceedings of the 2008 ACM symposium on Applied computing, March 16-20, 2008, Fortaleza, Ceara, Brazil [doi>10.1145/1363686.1363954]
• 15: Nima Taghipour , Ahmad Kardan , Saeed Shiry Ghidary, Usage-based web recommendations: a reinforcement learning approach, Proceedings of the 2007 ACM conference on Recommender systems, October 19-20, 2007, Minneapolis, MN, USA [doi>10.1145/1297231.1297250]
• 16: S. ten Hagen, M. van Someren, and V. Hollink. Exploration/exploitation in adaptive recommender systems. In Proceedings of the European Symposium on Intelligent Technologies, Hybrid Systems and their implementation on Smart Adaptive Systems, Oulu, Finland, 2003.
• 17: J. R. Tetreault and D. J. Litman. Using reinforcement learning to build a better model of dialogue state. In EACL, 2006.
• 18: C. A. Thompson, M. H. Goker, and P. Langley. A personalized system for conversational recommendations. Artificial Intelligence Research, 21:393--428, 2004.
• 19: Y. Z. Wei, L. Moreau, and N. R. Jennings. Learning users' interests in a market-based recommender system. In IDEAL, pages 833--840, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.2 Types of Systems
          Subjects: Decision support (e.g., MIS)

General Terms:
Algorithms, Design, Experimentation, Human Factors, Performance

Keywords:
adaptivity, conversational recommender systems, reinforcement learning, state features selection