Cross domain distribution adaptation via kernel mapping

ABSTRACT

When labeled examples are limited and difficult to obtain, transfer learning employs knowledge from a source domain to improve learning accuracy in the target domain. However, the assumption made by existing approaches, that the marginal and conditional probabilities are directly related between source and target domains, has limited applicability in either the original space or its linear transformations. To solve this problem, we propose an adaptive kernel approach that maps the marginal distribution of target-domain and source-domain data into a common kernel space, and utilize a sample selection strategy to draw conditional probabilities between the two domains closer. We formally show that under the kernel-mapping space, the difference in distributions between the two domains is bounded; and the prediction error of the proposed approach can also be bounded. Experimental results demonstrate that the proposed method outperforms both traditional inductive classifiers and the state-of-the-art boosting-based transfer algorithms on most domains, including text categorization and web page ratings. In particular, it can achieve around 10% higher accuracy than other approaches for the text categorization problem. The source code and datasets are available from the authors.

REFERENCES

• 1: M. Amini, F. Laviolette, and N. Usunier. A transductive bound for the voted classifier with an application to semi-supervised learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21. 2009.
• 2: A. Asuncion and D. J. Newman. UCI machine learning repository, 2007. http://www.ics.uci.edu/mlearn/MLRepository.html.
• 3: G. Baudat , F. Anouar, Generalized Discriminant Analysis Using a Kernel Approach, Neural Computation, v.12 n.10, p.2385-2404, October 2000 [doi>10.1162/089976600300014980]
• 4: Mikhail Belkin , Partha Niyogi , Vikas Sindhwani, Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples, The Journal of Machine Learning Research, 7, p.2399-2434, 12/1/2006
• 5: Steffen Bickel , Michael Brückner , Tobias Scheffer, Discriminative learning for differing training and test distributions, Proceedings of the 24th international conference on Machine learning, p.81-88, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273507]
• 6: J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman. Learning bounds for domain adaptation. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20. 2008.
• 7: F. Cucker and S. Smale. Best choices for regularization parameters in learning theory: On the bias-variance problem. Foundations of Computational Mathematics, 2(4):413--428, 2002.
• 8: Wenyuan Dai , Qiang Yang , Gui-Rong Xue , Yong Yu, Boosting for transfer learning, Proceedings of the 24th international conference on Machine learning, p.193-200, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273521]
• 9: Dmitry Davidov , Evgeniy Gabrilovich , Shaul Markovitch, Parameterized generation of labeled datasets for text categorization based on a hierarchical directory, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009036]
• 10: I. Davidson and W. Fan. When efficient model averaging out-performs boosting and bagging. In Knowledge Discovery in Databases: PKDD 2006, 10th European Conference on Principles and Practice of Knowledge Discovery in Databases, Berlin, Germany, September 18-22, 2006, Proceedings, pages 478--486. Springer, 2006.
• 11: W. Fan and I. Davidson. On sample selection bias and its efficient correction via model averaging and unlabeled examples. In Proceedings of the Seventh SIAM International Conference on Data Mining, SDM 2007, Minneapolis, Minnesota, USA, 2007. SIAM.
• 12: Jing Gao , Wei Fan , Jing Jiang , Jiawei Han, Knowledge transfer via multiple model local structure mapping, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA [doi>10.1145/1401890.1401928]
• 13: S. Y. Huang and C. R. Hwang. Kernel fisher's discriminant analysis in gaussian reproducing kernel hilbert space. Technical report, Institute of Statistical Science, Academia Sinica, Taiwan, 2005.
• 14: S. J. Pan, J. T. Kwok, and Q. Yang. Transfer learning via dimensionality reduction. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13-17, 2008, pages 677--682. AAAI Press, 2008.
• 15: S. J. Pan and Q. Yang. A survey on transfer learning. Technical Report HKUST-CS08-08, Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China, November 2008.
• 16: J. Ren, X. Shi, W. Fan, and P. S. Yu. Type-independent correction of sample selection bias via structural discovery and re-balancing. In Proceedings of the Eighth SIAM International Conference on Data Mining, SDM 2008, pages 565--576, Atlanta, Georgia, USA, 2008. SIAM.
• 17: Ulrich Rückert , Stefan Kramer, Kernel-Based Inductive Transfer, Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II, September 15-19, 2008, Antwerp, Belgium [doi>10.1007/978-3-540-87481-2_15]
• 18: Sandeepkumar Satpal , Sunita Sarawagi, Domain Adaptation of Conditional Probability Models Via Feature Subsetting, Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74976-9_23]
• 19: Bernhard Schölkopf , Ralf Herbrich , Alex J. Smola, A Generalized Representer Theorem, Proceedings of the 14th Annual Conference on Computational Learning Theory and and 5th European Conference on Computational Learning Theory, p.416-426, July 16-19, 2001
• 20: Zheng Wang , Yangqiu Song , Changshui Zhang, Transferred Dimensionality Reduction, Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II, September 15-19, 2008, Antwerp, Belgium [doi>10.1007/978-3-540-87481-2_36]
• 21: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 22: Keisuke Yamazaki , Motoaki Kawanabe , Sumio Watanabe , Masashi Sugiyama , Klaus-Robert Müller, Asymptotic Bayesian generalization error when training and test distributions are different, Proceedings of the 24th international conference on Machine learning, p.1079-1086, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273632]
• 23: Peng Zhang , Norbert Riedel, Discriminant Analysis: A Unified Approach, Proceedings of the Fifth IEEE International Conference on Data Mining, p.514-521, November 27-30, 2005 [doi>10.1109/ICDM.2005.51]
• 24: Xiaojin Zhu , John Lafferty , Ronald Rosenfeld, Semi-supervised learning with graphs, Carnegie Mellon University, Pittsburgh, PA, 2005

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Algorithms

Keywords:
domain transfer, ensemble, generalization bound, kernel