Regularized multi--task learning

ABSTRACT

Past empirical work has shown that learning multiple related tasks from data simultaneously can be advantageous in terms of predictive performance relative to learning these tasks independently. In this paper we present an approach to multi--task learning based on the minimization of regularization functionals similar to existing ones, such as the one for Support Vector Machines (SVMs), that have been successfully used in the past for single--task learning. Our approach allows to model the relation between tasks in terms of a novel kernel function that uses a task--coupling parameter. We implement an instance of the proposed approach similar to SVMs and test it empirically using simulated as well as real data. The experimental results show that the proposed method performs better than existing multi--task learning methods and largely outperforms single--task learning using SVMs.

REFERENCES

• 1: G.M. Allenby and P.E. Rossi. Marketing Models of Consumer Heterogeneity. Journal of Econometrics, 89, p. 57--78, 1999.
• 2: Neeraj Arora , Greg M. Allenby , James L. Ginter, A Hierarchical Bayes Model of Primary and Secondary Demand, Marketing Science, v.17 n.1, p.29-44, January 1998 [doi>10.1287/mksc.17.1.29]
• 3: N. Arora and J. Huber. Improving parameter estimates and model prediction by aggregate customization in choice experiments. Journal of Consumer Research, Vol. 28, September 2001.
• 4: Bart Bakker , Tom Heskes, Task clustering and gating for bayesian multitask learning, The Journal of Machine Learning Research, 4, p.83-99, 12/1/2003
• 5: Jonathan Baxter, A Bayesian/Information Theoretic Model of Learning to Learn viaMultiple Task Sampling, Machine Learning, v.28 n.1, p.7-39, July 1997 [doi>10.1023/A:1007327622663]
• 6: J. Baxter. A Model for Inductive Bias Learning. Journal of Artificial Intelligence Research, 12, p. 149--198, 2000.
• 7: Shai Ben-David , Johannes Gehrke , Reba Schuller, A theoretical framework for learning from a pool of disparate data sources, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775111]
• 8: S. Ben-David and R. Schuller. Exploiting Task Relatedness for Multiple Task Learning. Proceedings of Computational Learning Theory (COLT), 2003.
• 9: L. Breiman and J.H Friedman. Predicting Multivariate Responses in Multiple Linear Regression. Royal Statistical Society Series B, 1998.
• 10: P.J. Brown and J.V. Zidek. Adaptive Multivariate Ridge Regression. The Annals of Statistics, Vol. 8, No. 1, p. 64--74, 1980.
• 11: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 12: T. Evgeniou, C. Boussios, and G. Zacharia. Generalized Robust Conjoint Estimation. INSEAD Working Paper, 2002.
• 13: T. Evgeniou, M. Pontil, and T. Poggio. Regularization networks and support vector machines. Advances in Computational Mathematics, 13:1--50, 2000.
• 14: B. Heisele, T. Serre, M. Pontil, T. Vetter, and T. Poggio. Categorization by Learning and Combining Object Parts. In: Advances in Neural Information Processing Systems 14, Vancouver, Canada, Vol. 2, 1239--1245, 2002.
• 15: Tom Heskes, Empirical Bayes for Learning to Learn, Proceedings of the Seventeenth International Conference on Machine Learning, p.367-374, June 29-July 02, 2000
• 16: Michael I. Jordan , Robert A. Jacobs, Hierarchical mixtures of experts and the EM algorithm, Neural Computation, v.6 n.2, p.181-214, March 1994 [doi>10.1162/neco.1994.6.2.181]
• 17: Jaehwan Kim , Greg M. Allenby , Peter E. Rossi, Modeling Consumer Demand for Variety, Marketing Science, v.21 n.3, p.229-250, July 2002 [doi>10.1287/mksc.21.3.229.143]
• 18: G.R.G. Lanckriet, T. De Bie, N. Cristianini, M.I. Jordan, and W.S. Noble. A framework for genomic data fusion and its application to membrane protein prediction. Technical Report CSD--03--1273, Division of Computer Science, University of California, Berkeley, 2003.
• 19: O.L. Mangasarian. Nonlinear Programming. Classics in Applied Mathematics. SIAM, 1994.
• 20: C.A. Micchelli and M. Pontil. On Learning Vector--Valued Functions. Research Note RN/03/08, Dept of Computer Science, UCL, July 2003.
• 21: D.L. Silver and R.E Mercer. The parallel transfer of task knowledge using dynamic learning rates based on a measure of relatedness. Connection Science, 8, p. 277--294, 1996.
• 22: Sebastian Thrun , Lorien Pratt, Learning to learn, Kluwer Academic Publishers, Norwell, MA, 1998
• 23: Sebastian Thrun , Joseph O'Sullivan, Clustering learning tasks and the selective cross-task transfer of knowledge, Learning to learn, Kluwer Academic Publishers, Norwell, MA, 1998
• 24: O. Toubia, D.I. Simester, J.R. Hauser, and E. Dahan. Fast Polyhedral Adaptive Conjoint Estimation. Working paper, MIT Sloan School of Management, 2001.
• 25: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 26: G. Wahba. Splines Models for Observational Data. Series in Applied Mathematics, Vol. 59, SIAM, Philadelphia, 1990.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Performance

Keywords:
kernel methods, multi--task learning, regularization, support vector machines