Knowledge transfer via multiple model local structure mapping

ABSTRACT

The effectiveness of knowledge transfer using classification algorithms depends on the difference between the distribution that generates the training examples and the one from which test examples are to be drawn. The task can be especially difficult when the training examples are from one or several domains different from the test domain. In this paper, we propose a locally weighted ensemble framework to combine multiple models for transfer learning, where the weights are dynamically assigned according to a model's predictive power on each test example. It can integrate the advantages of various learning algorithms and the labeled information from multiple training domains into one unified classification model, which can then be applied on a different domain. Importantly, different from many previously proposed methods, none of the base learning method is required to be specifically designed for transfer learning. We show the optimality of a locally weighted ensemble framework as a general approach to combine multiple models for domain transfer. We then propose an implementation of the local weight assignments by mapping the structures of a model onto the structures of the test domain, and then weighting each model locally according to its consistency with the neighborhood structure around the test example. Experimental results on text classification, spam filtering and intrusion detection data sets demonstrate significant improvements in classification accuracy gained by the framework. On a transfer learning task of newsgroup message categorization, the proposed locally weighted ensemble framework achieves 97% accuracy when the best single model predicts correctly only on 73% of the test examples. In summary, the improvement in accuracy is over 10% and up to 30% across different problems.

REFERENCES

• 1: Christopher G. Atkeson , Andrew W. Moore , Stefan Schaal, Locally Weighted Learning, Artificial Intelligence Review, v.11 n.1-5, p.11-73, Feb. 1997 [doi>10.1023/A:1006559212014]
• 2: Eric Bauer , Ron Kohavi, An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants, Machine Learning, v.36 n.1-2, p.105-139, July-August 1999 [doi>10.1023/A:1007515423169]
• 3: S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In Proc. of NIPS' 07, pages 137--144. 2007.
• 4: Paul N. Bennett , Susan T. Dumais , Eric Horvitz, The Combination of Text Classifiers Using Reliability Indicators, Information Retrieval, v.8 n.1, p.67-100, January 2005 [doi>10.1023/B:INRT.0000048491.59134.94]
• 5: Steffen Bickel , Michael Brückner , Tobias Scheffer, Discriminative learning for differing training and test distributions, Proceedings of the 24th international conference on Machine learning, p.81-88, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273507]
• 6: A.J. Carlson, C.M. Cumby, J.L.R. Nicholas D.Rizzolo, and D.Roth. Snow learning architecture. http://l2r.cs.uiuc.edu/~cogcomp/asoftware.php?skey=SNOW#projects.
• 7: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 8: C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 9: Wenyuan Dai , Gui-Rong Xue , Qiang Yang , Yong Yu, Co-clustering based classification for out-of-domain documents, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281218]
• 10: Wenyuan Dai , Qiang Yang , Gui-Rong Xue , Yong Yu, Boosting for transfer learning, Proceedings of the 24th international conference on Machine learning, p.193-200, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273521]
• 11: H. Daumé and D. Marcu. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101--126, 2006.
• 12: Thomas G. Dietterich, Ensemble Methods in Machine Learning, Proceedings of the First International Workshop on Multiple Classifier Systems, p.1-15, June 21-23, 2000
• 13: Wei Fan, Systematic data selection to mine concept-drifting data streams, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014069]
• 14: W. Fan and I. Davidson. On sample selection bias and its efficient correction via model averaging and unlabeled examples. In Proc. of SDM'07.
• 15: Jing Gao , Wei Fan , Jiawei Han, On Appropriate Assumptions to Mine Data Streams: Analysis and Practice, Proceedings of the 2007 Seventh IEEE International Conference on Data Mining, p.143-152, October 28-31, 2007 [doi>10.1109/ICDM.2007.96]
• 16: A. Genkin, D. D. Lewis, and D. Madigan. Bbr: Bayesian logistic regression software. http://stat.rutgers.edu/~madigan/BBR/.
• 17: J. Hoeting, D. Madigan, A. Raftery, and C. Volinsky. Bayesian model averaging: a tutorial. Statist. Sci., 14:382--417, 1999.
• 18: J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and B. Scholkopf. Correcting sample selection bias by unlabeled data. In Proc. of NIPS' 06, pages 601--608. 2007.
• 19: Robert A. Jacobs , Michael I. Jordan , Steven J. Nowlan , Geoffrey E. Hinton, Adaptive mixtures of local experts, Neural Computation, v.3 n.1, p.79-87, Spring 1991 [doi>10.1162/neco.1991.3.1.79]
• 20: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 21: G. Karypis. Cluto - family of data clustering software tools. http://glaros.dtc.umn.edu/gkhome/views/cluto.
• 22: X. Li and J. Bilmes. A Bayesian divergence prior for classifier adaptation. In Proc. of AISTATS' 07, 2007.
• 23: D.M. Roy and L.P. Kaelbling. Efficient bayesian task-level transfer learning. In Proc. of IJCAI '07.
• 24: Sandeepkumar Satpal , Sunita Sarawagi, Domain Adaptation of Conditional Probability Models Via Feature Subsetting, Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74976-9_23]
• 25: H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227--244, 2000.
• 26: A. Storkey and M. Sugiyama. Mixture regression for covariate shift. In Proc. of NIPS' 06, pages 1337--1344.
• 27: Haixun Wang , Wei Fan , Philip S. Yu , Jiawei Han, Mining concept-drifting data streams using ensemble classifiers, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956778]
• 28: X. Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms

Keywords:
classification, ensemble, semi-supervised learning, transfer learning