Transfer learning from multiple source domains via consensus regularization

ABSTRACT

Recent years have witnessed an increased interest in transfer learning. Despite the vast amount of research performed in this field, there are remaining challenges in applying the knowledge learnt from multiple source domains to a target domain. First, data from multiple source domains can be semantically related, but have different distributions. It is not clear how to exploit the distribution differences among multiple source domains to boost the learning performance in a target domain. Second, many real-world applications demand this transfer learning to be performed in a distributed manner. To meet these challenges, we propose a consensus regularization framework for transfer learning from multiple source domains to a target domain. In this framework, a local classifier is trained by considering both local data available in a source domain and the prediction consensus with the classifiers from other source domains. In addition, the training algorithm can be implemented in a distributed manner, in which all the source-domains are treated as slave nodes and the target domain is used as the master node. To combine the training results from multiple source domains, it only needs share some statistical data rather than the full contents of their labeled data. This can modestly relieve the privacy concerns and avoid the need to upload all data to a central location. Finally, our experimental results show the effectiveness of our consensus regularization learning.

REFERENCES

• 1: Wenyuan Dai , Qiang Yang , Gui-Rong Xue , Yong Yu, Boosting for transfer learning, Proceedings of the 24th international conference on Machine learning, p.193-200, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273521]
• 2: Wenyuan Dai , Gui-Rong Xue , Qiang Yang , Yong Yu, Co-clustering based classification for out-of-domain documents, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281218]
• 3: Dikan Xing , Wenyuan Dai , Gui-Rong Xue , Yong Yu, Bridged Refinement for Transfer Learning, Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74976-9_31]
• 4: A. Smeaton and P. Over. Trecvid: Benchmarking the effectiveness of information retrieval tasks on digital video. In Proc. of the Intl. Conf. on Image and Video Retrieval, 2003.
• 5: Jun Yang , Rong Yan , Alexander G. Hauptmann, Cross-domain video concept detection using adaptive svms, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291276]
• 6: David Hosmer and Stanley Lemeshow. Applied Logistic Regression. Wiley, New York, 2000.
• 7: Andrzej Ruszczynski, Nonlinear Optimization, Princeton University Press, Princeton, NJ, 2006
• 8: Thorsten Joachims, Transductive Inference for Text Classification using Support Vector Machines, Proceedings of the Sixteenth International Conference on Machine Learning, p.200-209, June 27-30, 1999
• 9: T. Joachims. Transductive learning via spectral graph partitioning. In Proc. of the 20th ICML, 2003.
• 10: Xuejun Liao , Ya Xue , Lawrence Carin, Logistic regression with an auxiliary data source, Proceedings of the 22nd international conference on Machine learning, p.505-512, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102415]
• 11: Andrew T. Smith , Charles Elkan, Making generative classifiers robust to selection bias, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281263]
• 12: Rajat Raina , Andrew Y. Ng , Daphne Koller, Constructing informative priors using transfer learning, Proceedings of the 23rd international conference on Machine learning, p.713-720, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143934]
• 13: Rajat Raina , Alexis Battle , Honglak Lee , Benjamin Packer , Andrew Y. Ng, Self-taught learning: transfer learning from unlabeled data, Proceedings of the 24th international conference on Machine learning, p.759-766, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273592]
• 14: Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Proc. of the 19th NIPS, pages 529--536, 2005.
• 15: David Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Proceedings of the 33rd annual meeting on Association for Computational Linguistics, p.189-196, June 26-30, 1995, Cambridge, Massachusetts [doi>10.3115/981658.981684]
• 16: Avrim Blum , Tom Mitchell, Combining labeled and unlabeled data with co-training, Proceedings of the eleventh annual conference on Computational learning theory, p.92-100, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279962]
• 17: Yves Grandvalet , Florence d'Alché-Buc , Christophe Ambroise, Boosting Mixture Models for Semi-supervised Learning, Proceedings of the International Conference on Artificial Neural Networks, p.41-48, August 21-25, 2001
• 18: V. Sindhwani, P. Niyogi, and M. Belkin. A co-regularization approach to semi-supervised learning with multiple views. In Proc. ICML Workshop on Learning with Multiple Views, 2005.
• 19: S. Dasgupta, M. L. Littman, and D. A. McAllester. Pac generalization bounds for co-training. In Proc. of the 15th NIPS, pages 375--382, 2001.
• 20: Steven Abney, Bootstrapping, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania [doi>10.3115/1073083.1073143]
• 21: Steven Abney, Understanding the Yarowsky Algorithm, Computational Linguistics, v.30 n.3, p.365-395, September 2004 [doi>10.1162/0891201041850876]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Induction

General Terms:
Algorithms, Experimentation, Performance

Keywords:
classification, consensus regularization, transfer learning