Rate it again: increasing recommendation accuracy by user re-rating

ABSTRACT

A common approach to designing Recommender Systems (RS) consists of asking users to explicitly rate items in order to collect feedback about their preferences. However, users have been shown to be inconsistent and to introduce a non-negligible amount of natural noise in their ratings that affects the accuracy of the predictions. In this paper, we present a novel approach to improve RS accuracy by reducing the natural noise in the input data via a preprocessing step. In order to quantitatively understand the impact of natural noise, we first analyze the response of common recommendation algorithms to this noise. Next, we propose a novel algorithm to denoise existing datasets by means of re-rating: i.e. by asking users to rate previously rated items again. This denoising step yields very significant accuracy improvements. However, re-rating all items in the original dataset is unpractical. Therefore, we study the accuracy gains obtained when re-rating only some of the ratings.In particular, we propose two partial denoising strategies: data and user-dependent denoising. Finally, we compare the value of adding a rating of an unseen item vs. re-rating an item. We conclude with a proposal for RS to improve the quality of their user data and hence their accuracy: asking users to re-rate items might, in some circumstances, be more beneficial than asking users to rate unseen items.

REFERENCES

• 1: G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. on Knowl. and Data Eng., 17(6):734--749, 2005.
• 2: X. Amatriain, J. Pujol, and N. Oliver. I like it... i like it not: Evaluating user ratings noise in recommender systems. In Proc. of UMAP'09, 2009.
• 3: C. Anderson. The Long Tail: Why the Future of Business is Selling Less of More. Hyperion, 2006.
• 4: R. M. Bell and Y. Koren. Improved neighborhood-based collaborative filtering. In Proc. of KDD Cup, 2007.
• 5: J. Bennet and S. Lanning. The netflix prize. In Proc. of KDD Work. on Large-scale Rec.. Sys., 2007.
• 6: S. Berkovsky, Y. Eytani, T. Kuflik, and F. Ricci. Enhancing privacy and preserving accuracy of a distributed collaborative filtering. In Proc. of Recsys '07, 2007.
• 7: D. Cosley, S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. Is seeing believing? how recommender system interfaces affect users' opinions. In Proc. of CHI'03, pages 585--592, 2003.
• 8: E. A. Greenleaf. Measuring extreme response style. The Public Opinion Quarterly, 56:328--351, 1992.
• 9: M. Harper, X. Li, Y. Chen, and J. Konstan. An economic model of user rating in an online recommender system. In Proc. of UM 05, 2005.
• 10: J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Trans. on Inf. Syst., 22(1):5--53, 2004.
• 11: W. Hill, L. Stead, M. Rosenstein, and G. Furnas. Recommending and evaluating choices in a virtual community of use. In Proc. of CHI '95, pages 194--201, 1995.
• 12: D. W. Oard and J. Kim. Implicit feedback for recommender systems. In AAAI Works. on Rec. Sys., 1998.
• 13: M. P. O'Mahony. Detecting noise in recommender system databases. In Proc. of IUI'06, pages 109--115, 2006.
• 14: A. Paterek. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of KDD Cup and Workshop 2007, 2007.
• 15: R. Rafter, M. O'Mahony, N.J.Hurley, and B.Smyth. What have the neighbours ever done for us? a collaborative filtering perspective. In Proc. of UMAP '09, 2009.
• 16: A. M. Rashid, I. Albert, D. Cosley, S. K. Lam, S. M. Mcnee, J. A. Konstan, and J. Riedl. Getting to know you: Learning new user preferences in recommender systems. In Proc. of IUI '02, 2002.
• 17: G. Torkzadeh and W. J. Doll. The test-retest reliability of user involvement instruments. Inf. Manag., 26(1):21--31, 1994.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Performance, Theory

Keywords:
collaborative filtering, noise, recommender systems