Spectral domain-transfer learning

ABSTRACT

Traditional spectral classification has been proved to be effective in dealing with both labeled and unlabeled data when these data are from the same domain. In many real world applications, however, we wish to make use of the labeled data from one domain (called in-domain) to classify the unlabeled data in a different domain (out-of-domain). This problem often happens when obtaining labeled data in one domain is difficult while there are plenty of labeled data from a related but different domain. In general, this is a transfer learning problem where we wish to classify the unlabeled data through the labeled data even though these data are not from the same domain. In this paper, we formulate this domain-transfer learning problem under a novel spectral classification framework, where the objective function is introduced to seek consistency between the in-domain supervision and the out-of-domain intrinsic structure. Through optimization of the cost function, the label information from the in-domain data is effectively transferred to help classify the unlabeled data from the out-of-domain. We conduct extensive experiments to evaluate our method and show that our algorithm achieves significant improvements on classification performance over many state-of-the-art algorithms.

REFERENCES

• 1: S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In NIPS, 2007.
• 2: S. Ben-David and R. Schuller. Exploiting task relatedness for multiple task learning. In COLT, 2003.
• 3: Steffen Bickel , Michael Brückner , Tobias Scheffer, Discriminative learning for differing training and test distributions, Proceedings of the 24th international conference on Machine learning, p.81-88, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273507]
• 4: S. Bickel and T. Scheffer. Dirichlet-enhanced spam filtering based on biased samples. In NIPS, 2007.
• 5: J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman. Learning bounds for domain adaptation. In NIPS, 2008.
• 6: J. Blitzer, R. McDonald, and F. Pereira. Domain adaptation with structural correspondence learning. In EMNLP, 2006.
• 7: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 8: C.-K. Cheng and Y.-C. A. Wei. An improved two-way partitioning algorithm with stable performance {VLSI}. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 10(12):1502--1511, 1991.
• 9: F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
• 10: Wenyuan Dai , Gui-Rong Xue , Qiang Yang , Yong Yu, Co-clustering based classification for out-of-domain documents, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281218]
• 11: W. Dai, G.-R. Xue, Q. Yang, and Y. Yu. Transferring naive bayes classifiers for text classification. In AAAI, 2007.
• 12: Wenyuan Dai , Qiang Yang , Gui-Rong Xue , Yong Yu, Boosting for transfer learning, Proceedings of the 24th international conference on Machine learning, p.193-200, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273521]
• 13: H. Daum2e III. Frustratingly easy domain adaptation. In ACL, 2007.
• 14: H. Daume III and D. Marcu. Domain adaptation for statistical classifiers. JAIR, 1:1--15, 2006.
• 15: Chris H. Q. Ding , Xiaofeng He , Hongyuan Zha , Ming Gu , Horst D. Simon, A Min-max Cut Algorithm for Graph Partitioning and Data Clustering, Proceedings of the 2001 IEEE International Conference on Data Mining, p.107-114, November 29-December 02, 2001
• 16: G. H. Golub and C. F. Van Loan. Matrix Computation. The Johns Hopkins University Press Baltimore, 1996.
• 17: J. J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153--162, 1979.
• 18: J. Huang, A. J. Smola, A. Gretton, K. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. In NIPS, 2007.
• 19: Xiang Ji , Wei Xu, Document clustering with prior knowledge, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148241]
• 20: Thorsten Joachims, Transductive Inference for Text Classification using Support Vector Machines, Proceedings of the Sixteenth International Conference on Machine Learning, p.200-209, June 27-30, 1999
• 21: T. Joachims. Transductive learning via spectral graph partitioning. In ICML, 2003.
• 22: S. D. Kamvar, D. Klein, and C. D. Manning. Spectral learning. In IJCAI, 2003.
• 23: Xuejun Liao , Ya Xue , Lawrence Carin, Logistic regression with an auxiliary data source, Proceedings of the 22nd international conference on Machine learning, p.505-512, August 07-11, 2005, Bonn, Germany [doi>10.1145/1102351.1102415]
• 24: M. Meila and J. Shi. A random walks view of spectral segmentation. In Proceedings of the 8th International Workshop on Artificial Intelligence and Statistics, 2001.
• 25: A. Y. Ng, M. I. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, 2001.
• 26: M. Porter. An algorithm for suffix stripping program. Program, 14(3):130--137, 1980.
• 27: J. Schmidhuber. On learning how to learn learning strategies. Technical Report FKI-198-94, Fakultat fur Informatik, 1994.
• 28: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688]
• 29: H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227--244, 2000.
• 30: S. Thrun and T. Mitchell. Learning one more thing. In IJCAI, 1995.
• 31: Kiri Wagstaff , Claire Cardie, Clustering with Instance-level Constraints, Proceedings of the Seventeenth International Conference on Machine Learning, p.1103-1110, June 29-July 02, 2000
• 32: Pengcheng Wu , Thomas G. Dietterich, Improving SVM accuracy by training on auxiliary data sources, Proceedings of the twenty-first international conference on Machine learning, p.110, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015436]
• 33: Dikan Xing , Wenyuan Dai , Gui-Rong Xue , Yong Yu, Bridged Refinement for Transfer Learning, Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74976-9_31]
• 34: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997
• 35: Bianca Zadrozny, Learning and evaluating classifiers under sample selection bias, Proceedings of the twenty-first international conference on Machine learning, p.114, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015425]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Algorithms, Experimentation

Keywords:
spectral learning, transfer learning