Bayesian probabilistic matrix factorization using Markov chain Monte Carlo

ABSTRACT

Low-rank matrix approximation methods provide one of the simplest and most effective approaches to collaborative filtering. Such models are usually fitted to data by finding a MAP estimate of the model parameters, a procedure that can be performed efficiently even on very large datasets. However, unless the regularization parameters are tuned carefully, this approach is prone to overfitting because it finds a single point estimate of the parameters. In this paper we present a fully Bayesian treatment of the Probabilistic Matrix Factorization (PMF) model in which model capacity is controlled automatically by integrating over all model parameters and hyperparameters. We show that Bayesian PMF models can be efficiently trained using Markov chain Monte Carlo methods by applying them to the Netflix dataset, which consists of over 100 million movie ratings. The resulting models achieve significantly higher prediction accuracy than PMF models trained using MAP estimation.

REFERENCES

• 1: Geoffrey E. Hinton , Drew van Camp, Keeping the neural networks simple by minimizing the description length of the weights, Proceedings of the sixth annual conference on Computational learning theory, p.5-13, July 26-28, 1993, Santa Cruz, California, United States [doi>10.1145/168304.168306]
• 2: Hofmann, T. (1999). Probabilistic latent semantic analysis. Proceedings of the 15th Conference on Uncertainty in AI (pp. 289--296). San Fransisco, California: Morgan Kaufmann.
• 3: Michael I. Jordan , Zoubin Ghahramani , Tommi S. Jaakkola , Lawrence K. Saul, An Introduction to Variational Methods for Graphical Models, Machine Learning, v.37 n.2, p.183-233, Nov.1.1999 [doi>10.1023/A:1007665907178]
• 4: Lim, Y. J., & Teh, Y. W. (2007). Variational Bayesian approach to movie rating prediction. Proceedings of KDD Cup and Workshop.
• 5: Marlin, B. (2004). Modeling user rating profiles for collaborative filtering. In S. Thrun, L. Saul and B. Schölkopf (Eds.), Advances in neural information processing systems 16. Cambridge, MA: MIT Press.
• 6: Benjamin Marlin , Richard S. Zemel, The multiple multiplicative factor model for collaborative filtering, Proceedings of the twenty-first international conference on Machine learning, p.73, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015437]
• 7: Neal, R. M. (1993). Probabilistic inference using Markov chain Monte Carlo methods (Technical Report CRG-TR-93-1). Department of Computer Science, University of Toronto.
• 8: Steven J. Nowlan , Geoffrey E. Hinton, Simplifying neural networks by soft weight-sharing, Neural Computation, v.4 n.4, p.473-493, July 1992 [doi>10.1162/neco.1992.4.4.473]
• 9: Tapani Raiko , Alexander Ilin , Juha Karhunen, Principal Component Analysis for Large Scale Problems with Lots of Missing Values, Proceedings of the 18th European conference on Machine Learning, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74958-5_69]
• 10: Jasson D. M. Rennie , Nathan Srebro, Fast maximum margin matrix factorization for collaborative prediction, Proceedings of the 22nd international conference on Machine learning, p.713-719, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102441]
• 11: Salakhutdinov, R., & Mnih, A. (2008). Probabilistic matrix factorization. Advances in Neural Information Processing Systems 20. Cambridge, MA: MIT Press.
• 12: Srebro, N., & Jaakkola, T. (2003). Weighted low-rank approximations. Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), Washington, DC, USA (pp. 720--727). AAAI Press.

INDEX TERMS

Primary Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.1 Numerical Algorithms and Problems
          Subjects: Computations on matrices

Additional Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.1 Numerical Algorithms and Problems
          Subjects: Computations in finite fields; Number-theoretic computations (e.g., factoring, primality testing)

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo); Markov processes

General Terms:
Algorithms, Design, Theory