Logistic regression with an auxiliary data source

ABSTRACT

To achieve good generalization in supervised learning, the training and testing examples are usually required to be drawn from the same source distribution. In this paper we propose a method to relax this requirement in the context of logistic regression. Assuming D^p and D^a are two sets of examples drawn from two mismatched distributions, where D^a are fully labeled and D^p partially labeled, our objective is to complete the labels of D^p. We introduce an auxiliary variable μ for each example in D^a to reflect its mismatch with D^p. Under an appropriate constraint the μ's are estimated as a byproduct, along with the classifier. We also present an active learning approach for selecting the labeled examples in D^p. The proposed algorithm, called "Migratory-Logit" or M-Logit, is demonstrated successfully on simulated as well as real data sets.

REFERENCES

• 1: Bertsekas, D. P. (1999). Nonlinear programming (2nd edition). Athena Scientific.
• 2: Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1995). Active learning with statistical models. Advances in Neural Information Processing Systems, 7, 705--712.
• 3: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 4: Fedorov, V. V. (1972). Theory of optimal experiments. Academic Press.
• 5: Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153--161.
• 6: Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation, and active learning. Advances in Neural Information Processing Systems, 7, 231--238.
• 7: David J. C. MacKay, Information-based objective functions for active data selection, Neural Computation, v.4 n.4, p.590-604, July 1992 [doi>10.1162/neco.1992.4.4.590]
• 8: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 9: Pengcheng Wu , Thomas G. Dietterich, Improving SVM accuracy by training on auxiliary data sources, Proceedings of the twenty-first international conference on Machine learning, p.110, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015436]
• 10: Bianca Zadrozny, Learning and evaluating classifiers under sample selection bias, Proceedings of the twenty-first international conference on Machine learning, p.114, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015425]
• 11: Bianca Zadrozny , John Langford , Naoki Abe, Cost-Sensitive Learning by Cost-Proportionate Example Weighting, Proceedings of the Third IEEE International Conference on Data Mining, p.435, November 19-22, 2003