Large-scale collaborative prediction using a nonparametric random effects model

ABSTRACT

A nonparametric model is introduced that allows multiple related regression tasks to take inputs from a common data space. Traditional transfer learning models can be inappropriate if the dependence among the outputs cannot be fully resolved by known input-specific and task-specific predictors. The proposed model treats such output responses as conditionally independent, given known predictors and appropriate unobserved random effects. The model is nonparametric in the sense that the dimensionality of random effects is not specified a priori but is instead determined from data. An approach to estimating the model is presented uses an EM algorithm that is efficient on a very large scale collaborative prediction problem. The obtained prediction accuracy is competitive with state-of-the-art results.

REFERENCES

• 1: Bell, R. M., Koren, Y., & Volinsky, C. (2007). The BellKor solution to the Netflix prize (Technical Report). AT&T Labs.
• 2: Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008). Multi-task Gaussian process prediction. Advances in Neural Information Processing Systems 20 (NIPS), 153--160.
• 3: Dawid, A. P. (1981). Some matrix-variate distribution theory: notational considerations and a Bayesian application. Biometrika, 68, 265--274.
• 4: Gupta, A. K., & Naga, D. K. (1999). Matrix variate distributions. Chapman & Hall/CRC.
• 5: Hoff, P. (2005). Bilinear mixed-effects models for dyadic data. Journal of the American Statistical Assosciation, 100, 286--295.
• 6: Kurucz, M., Benczur, A. A., & Csalogany, K. (2007). Methods for large scale SVD with missing values. Proceedings of KDD Cup and Workshop.
• 7: Neil D. Lawrence , John C. Platt, Learning to learn with the informative vector machine, Proceedings of the twenty-first international conference on Machine learning, p.65, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015382]
• 8: Lim, Y. J., & Teh, Y. W. (2007). Variational Bayesian approach to movie rating prediction. Proceedings of KDD Cup and Workshop.
• 9: Jasson D. M. Rennie , Nathan Srebro, Fast maximum margin matrix factorization for collaborative prediction, Proceedings of the 22nd international conference on Machine learning, p.713-719, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102441]
• 10: Ruslan Salakhutdinov , Andriy Mnih, Bayesian probabilistic matrix factorization using Markov chain Monte Carlo, Proceedings of the 25th international conference on Machine learning, p.880-887, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390267]
• 11: Salakhutdinov, R., & Mnih, A. (2008b). Probabilistic matrix factorization. Advances in Neural Information Processing Systems 20 (NIPS), 1257--1264.
• 12: Ruslan Salakhutdinov , Andriy Mnih , Geoffrey Hinton, Restricted Boltzmann machines for collaborative filtering, Proceedings of the 24th international conference on Machine learning, p.791-798, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273596]
• 13: Schwaighofer, A., Tresp, V., & Yu, K. (2005). Hierarchical Bayesian modelling with Gaussian processes. Advances in Neural Information Processing Systems 17 (NIPS), 1209--1216.
• 14: Srebro, N., Rennie, J. D. M., & Jaakola, T. S. (2005). Maximum-margin matrix factorization. Advances in Neural Information Processing Systems 18 (NIPS), 1329--1336.
• 15: Teh, Y., Seeger, M., & Jordan, M. (2005). Semiparametric latent factor models. The 8th Conference on Artificial Intelligence and Statistics (AISTATS).
• 16: Tipping, M. E., & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statisitical Scoiety, B, 611--622.
• 17: Yu, K., Chu, W., Yu, S., Tresp, V., & Xu, Z. (2007). Stochastic relational models for discriminative link prediction. Advances in Neural Information Processing Systems 19 (NIPS), 1553--1560.
• 18: Kai Yu , Volker Tresp , Anton Schwaighofer, Learning Gaussian processes from multiple tasks, Proceedings of the 22nd international conference on Machine learning, p.1012-1019, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102479]
• 19: Yu, K., Zhu, S., Lafferty, J., & Gong, Y. (2009). Fast nonparametric matrix factorization for large-scale collaborative filtering. The 32nd SIGIR conference.
• 20: Shipeng Yu , Kai Yu , Volker Tresp , Hans-Peter Kriegel, Collaborative ordinal regression, Proceedings of the 23rd international conference on Machine learning, p.1089-1096, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143981]
• 21: Zhu, S., Yu, K., & Gong, Y. (2009). Stochastic relational models for large-scale dyadic data using MCMC. Advances in Neural Information Processing Systems 21 (NIPS), 1993--2000.

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Nonparametric statistics

Additional Classification:
  G. Mathematics of Computing
  G.2 DISCRETE MATHEMATICS
      G.2.1 Combinatorics
          Subjects: Combinatorial algorithms

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory