Constructing informative priors using transfer learning

ABSTRACT

Many applications of supervised learning require good generalization from limited labeled data. In the Bayesian setting, we can try to achieve this goal by using an informative prior over the parameters, one that encodes useful domain knowledge. Focusing on logistic regression, we present an algorithm for automatically constructing a multivariate Gaussian prior with a full covariance matrix for a given supervised learning task. This prior relaxes a commonly used but overly simplistic independence assumption, and allows parameters to be dependent. The algorithm uses other "similar" learning problems to estimate the covariance of pairs of individual parameters. We then use a semidefinite program to combine these estimates and learn a good prior for the current learning task. We apply our methods to binary text classification, and demonstrate a 20 to 40% test error reduction over a commonly used prior.

REFERENCES

• 1: Ando, R. K., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6, 1817--1853.
• 2: Jonathan Baxter, A Bayesian/Information Theoretic Model of Learning to Learn viaMultiple Task Sampling, Machine Learning, v.28 n.1, p.7-39, July 1997 [doi>10.1023/A:1007327622663]
• 3: Ben-David, S., & Schuller, R. (2003). Exploiting task relatedness for multiple task learning. COLT.
• 4: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 5: Chung, F. (1997). Spectral graph theory. Regional Conference Series in Mathematics, American Mathematical Society, 92, 1--212.
• 6: Efron, B. (1979). Bootstrap methods: Another look at the jackknife. In The Annals of Statistics, vol. 7, 1--26.
• 7: Lang, K. (1995). Newsweeder: learning to filter net-news. ICML.
• 8: Neil D. Lawrence , John C. Platt, Learning to learn with the informative vector machine, Proceedings of the twenty-first international conference on Machine learning, p.65, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015382]
• 9: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 10: Ng, A. Y., Jordan, M., & Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. NIPS.
• 11: Nigam, K., Lafferty, J., & McCallum, A. (1999). Using maximum entropy for text classification. IJ-CAI Workshop on Machine Learning for Information Filtering.
• 12: Thrun, S. (1996). Is learning the n-th thing any easier than learning the first? NIPS.
• 13: Kai Yu , Volker Tresp , Anton Schwaighofer, Learning Gaussian processes from multiple tasks, Proceedings of the 22nd international conference on Machine learning, p.1012-1019, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102479]

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
      I.5.4 Applications
          Subjects: Text processing

General Terms:
Algorithms