Making generative classifiers robust to selection bias

ABSTRACT

This paper presents approaches to semi-supervised learning when the labeled training data and test data are differently distributed. Specifically, the samples selected for labeling are a biased subset of some general distribution and the test set consists of samples drawn from either that general distribution or the distribution of the unlabeled samples. An example of the former appears in loan application approval, where samples with repay/default labels exist only for approved applicants and the goal is to model the repay/default behavior of all applicants. An example of the latter appears in spam filtering, in which the labeled samples can be out-dated due to the cost of labeling email by hand, but an unlabeled set of up-to-date emails exists and the goal is to build a filter to sort new incoming email.Most approaches to overcoming such bias in the literature rely on the assumption that samples are selected for labeling depending only on the features, not the labels, a case in which provably correct methods exist. The missing labels are said to be "missing at random" (MAR). In real applications, however, the selection bias can be more severe. When the MAR conditional independence assumption is not satisfied and missing labels are said to be "missing not at random" (MNAR), and no learning method is provably always correct.We present a generative classifier, the shifted mixture model (SMM), with separate representations of the distributions of the labeled samples and the unlabeled samples. The SMM makes no conditional independence assumptions and can model distributions of semi-labeled data sets with arbitrary bias in the labeling. We present a learning method based on the expectation maximization (EM) algorithm that, while not always able to overcome arbitrary labeling bias, learns SMMs with higher test-set accuracy in real-world data sets (with MNAR bias) than existing learning methods that are proven to overcome MAR bias.

REFERENCES

• 1: K. Benson and A. J. Hartz. A comparison of observational studies and randomized controlled trials. The New England Journal of Medicine, 342(25): 1878--1886, 2000.
• 2: Steffen Bickel , Michael Brückner , Tobias Scheffer, Discriminative learning for differing training and test distributions, Proceedings of the 24th international conference on Machine learning, p.81-88, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273507]
• 3: S. Bickel and T. Scheffer. Dirichlet-enhanced spam filtering based on biased samples. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 161--168. MIT Press, Cambridge, MA, 2007.
• 4: W. J. Boyes, D. J. Hoffman, and S. A. Low. An econometric analysis of the bank credit scoring problem. Journal of Econometrics, 40(1):3--14, 1989.
• 5: C. Chelba and A. Acero. Adaptation of maximum entropy capitalizer: Little data can help a lot. Computer Speech & Language, 20(4):382--399, 2006.
• 6: G. Chen and T. Astebro. The economic value of reject inference in credit scoring. In J. N. C. L. C. Thomas and D. B. Edelman, editors, Credit Scoring and Credit Control VII: Proceedings of Conference held at University of Edinburgh, Edinburgh, Scotland, 5--7 September, 2001.
• 7: D. A. Cobb-Clark and T. Crossley. Econometrics for evaluations: An introduction to recent developments. The Economic Record, 79(247):491--511, 2003.
• 8: J. Crook and J. Banasik. Does reject inference really improve the performance of application scoring models? Technical Report Working Paper Series No. 02/3, Credit Research Centre, 2002.
• 9: A. Dempster, N. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39:1--38, 1977.
• 10: A. J. Feelders. An overview of model based reject inference for credit scoring. Technical report, Utrecht University, Institute for Information and Computing Sciences, (unpublished). http://www.cs.uu.nl/people/ad/mbrejinf.pdf.
• 11: P W. Glynn , D. L. Iglehart, Importance sampling for stochastic simulations, Management Science, v.35 n.11, p.1367-1392, Nov. 1989 [doi>10.1287/mnsc.35.11.1367]
• 12: J. Huang, A. Smola, A. Gretton, K. M. Borgwardt, and B. Schölkopf. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19. MIT Press, Cambridge, MA, 2007.
• 13: Roderick J A Little , Donald B Rubin, Statistical analysis with missing data, John Wiley & Sons, Inc., New York, NY, 1986
• 14: Kevin Patrick Murphy , Stuart Russell, Dynamic bayesian networks: representation, inference and learning, University of California, Berkeley, 2002
• 15: R. Pace and R. Barry. Sparse spatial autoregressions. Statistics and Probability Letters, 33:291--297, 1997.
• 16: J. Pearl. Graphical models for probabilistic and causal reasoning. In D. M. Gabbay and P. Smets, editors, Handbook of Defeasible Reasoning and Uncertainty Management Systems, Volume 1: Quantified Representation of Uncertainty and Imprecision, pages 367--389. Kluwer Academic Publishers, Dordrecht, 1998.
• 17: J. M. Robins, M. A. Hernan, and B. Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, 11(5):550--560, 2000.
• 18: P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41--56, 1983.
• 19: S. Rosset, J. Zhu, H. Zou, and T. Hastie. A method for inferring label sampling mechanisms in semi-supervised learning. Advances in Neural Information Processing Systems, 17: 1161--1168, 2005.
• 20: H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2): 227--244, 2000.
• 21: Andrew Smith , Charles Elkan, A Bayesian network framework for reject inference, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014085]
• 22: A. Storkey and M. Sugiyama. Mixture regression for covariate shift. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 1337--1344. MIT Press, Cambridge, MA, 2007.
• 23: M. Sugiyama and K.-R. Möller. Model selection under covariate shift. In W. Duch, J. Kacprzyk, E. Oja, and S. Zadrozny, editors, Artificial Neural Networks: Formal Models and Their Applications, volume 3697 of Lecture Notes in Computer Science, pages 235--240, Berlin, 2005. Springer.
• 24: A. J. Treno, P. J. Gruenewald, and F. W. Johnson. Sample selection bias in the emergency room: an examination of the role of alcohol in injury. Addiction, 93(1): 113--29, 1998.
• 25: Keisuke Yamazaki , Motoaki Kawanabe , Sumio Watanabe , Masashi Sugiyama , Klaus-Robert Müller, Asymptotic Bayesian generalization error when training and test distributions are different, Proceedings of the 24th international conference on Machine learning, p.1079-1086, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273632]
• 26: Bianca Zadrozny, Learning and evaluating classifiers under sample selection bias, Proceedings of the twenty-first international conference on Machine learning, p.114, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015425]
• 27: Bianca Zadrozny , Charles Elkan, Transforming classifier scores into accurate multiclass probability estimates, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775151]

INDEX TERMS

Primary Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Parameter learning

General Terms:
Algorithms, Economics, Theory

Keywords:
generative classifiers, reject inference, sample selection bias, semi-supervised learning