Restricted Boltzmann machines for collaborative filtering

ABSTRACT

Most of the existing approaches to collaborative filtering cannot handle very large data sets. In this paper we show how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies. We present efficient learning and inference procedures for this class of models and demonstrate that RBM's can be successfully applied to the Netflix data set, containing over 100 million user/movie ratings. We also show that RBM's slightly outperform carefully-tuned SVD models. When the predictions of multiple RBM models and multiple SVD models are linearly combined, we achieve an error rate that is well over 6% better than the score of Netflix's own system.

REFERENCES

• 1: John Canny, Collaborative filtering with privacy via factor analysis, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564419]
• 2: Carreira-Perpinan, M., & Hinton, G. (2005). On contrastive divergence learning. 10th Int. Work-shop on Artificial Intelligence and Statistics (AISTATS'2005).
• 3: Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A. (1990). Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41, 391--407.
• 4: Hinton, & Salakhutdinov (2006). Reducing the dimensionality of data with neural networks. Science, 313.
• 5: Geoffrey E. Hinton, Training products of experts by minimizing contrastive divergence, Neural Computation, v.14 n.8, p.1771-1800, August 2002 [doi>10.1162/089976602760128018]
• 6: Geoffrey E. Hinton , Simon Osindero , Yee-Whye Teh, A fast learning algorithm for deep belief nets, Neural Computation, v.18 n.7, p.1527-1554, July 2006 [doi>10.1162/neco.2006.18.7.1527]
• 7: Hofmann, T. (1999). Probabilistic latent semantic analysis. Proceedings of the 15th Conference on Uncertainty in AI (pp. 289--296). San Fransisco, California: Morgan Kaufmann.
• 8: Benjamin Marlin , Richard S. Zemel, The multiple multiplicative factor model for collaborative filtering, Proceedings of the twenty-first international conference on Machine learning, p.73, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015437]
• 9: Neal, R. M. (1993). Probabilistic inference using Markov chain Monte Carlo methods (Technical Report CRG-TR-93-1). Department of Computer Science, University of Toronto.
• 10: Salakhutdinov, R., & Hinton, G. E. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. AI and Statistics.
• 11: Srebro, N., & Jaakkola, T. (2003). Weighted low-rank approximations. Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21--24, 2003, Washington, DC, USA (pp. 720--727). AAAI Press.
• 12: Nathan Srebro , Tommi S. Jaakkola, Learning with matrix factorizations, Massachusetts Institute of Technology, Cambridge, MA, 2004
• 13: Sutskever, I., & Hinton, G. E. (2006). Learning multilevel distributed representations for high-dimensional sequences (Technical Report UTML TR 2006-003). Dept. of Computer Science, University of Toronto.
• 14: Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2006). Modeling human motion using binary latent variables. Advances in Neural Information Processing Systems. MIT Press.
• 15: Welling, M., Rosen-Zvi, M., & Hinton, G. (2005). Exponential family harmoniums with an application to information retrieval. NIPS 17 (pp. 1481--1488). Cambridge, MA: MIT Press.