A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems

ABSTRACT

Categorical data fields characterized by a large number of distinct values represent a serious challenge for many classification and regression algorithms that require numerical inputs. On the other hand, these types of data fields are quite common in real-world data mining applications and often contain potentially relevant information that is difficult to represent for modeling purposes.This paper presents a simple preprocessing scheme for high-cardinality categorical data that allows this class of attributes to be used in predictive models such as neural networks, linear and logistic regression. The proposed method is based on a well-established statistical method (empirical Bayes) that is straightforward to implement as an in-database procedure. Furthermore, for categorical attributes with an inherent hierarchical structure, like ZIP codes, the preprocessing scheme can directly leverage the hierarchy by blending statistics at the various levels of aggregation.While the statistical methods discussed in this paper were first introduced in the mid 1950's, the use of these methods as a preprocessing step for complex models, like neural networks, has not been previously discussed in any literature.

REFERENCES

• 1: Jonathan D. Becher , Pavel Berkhin , Edmund Freeman, Automating exploratory data analysis for efficient data mining, Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, p.424-429, August 20-23, 2000, Boston, Massachusetts, United States [doi>10.1145/347090.347179]
• 2: Carlin, B. P. and Louis T. A. Bayes and Empirical Bayes Methods for Data Analysis, New York, Chapman & Hall, 1996
• 3: Bojan Cestnik , Ivan Bratko, On estimating probabilities in tree pruning, Proceedings of the European working session on learning on Machine learning, p.138-150, March 1991, Porto, Portugal
• 4: Cestnik B., Estimating Probabilities: A Crucial Task in Machine Learning, Proc. of the 9th European Conf. on Artificial Intelligence, ECAI'90, 147-149, 1990
• 5: Gnanadesikan, R., Methods for Statistical Data Analysis of Multivariate Observations, Wiley, New York, 1977
• 6: Good, L. J., Probability and the weighting of evidence, London, Charles Griffing & Company Limited, 1950
• 7: http://www.unica-usa.com
• 8: Johnson, S. C. Hierarchical Clustering Schemes, Psychometrika, 2:241-254, 1967
• 9: Andrew McCallum , Ronald Rosenfeld , Tom M. Mitchell , Andrew Y. Ng, Improving Text Classification by Shrinkage in a Hierarchy of Classes, Proceedings of the Fifteenth International Conference on Machine Learning, p.359-367, July 24-27, 1998
• 10: Nishisato, S. Analysis of Categorical Data: Dual Scaling and Its Applications, Toronto: Toronto University Press, 1980
• 11: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 12: J. R. Quinlan, Induction of Decision Trees, Machine Learning, v.1 n.1, p.81-106 [doi>10.1023/A:1022643204877]
• 13: Robbins, H. An empirical Bayes approach to statistics, In Proc. 3rd Berkeley Symposium on Math Statistics and Probability, 1, Berkeley, CA: University of California Press, 157-164, 1955

INDEX TERMS

Keywords:
categorical attributes, empirical bayes, hierarchical attributes, neural networks, predictive models