Magical thinking in data mining: lessons from CoIL challenge 2000

ABSTRACT

CoIL challenge 2000 was a supervised learning contest that attracted 43 entries. The authors of 29 entries later wrote explanations of their work. This paper discusses these reports and reaches three main conclusions. First, naive Bayesian classifiers remain competitive in practice: they were used by both the winning entry and the next best entry. Second, identifying feature interactions correctly is important for maximizing predictive accuracy: this was the difference between the winning classifier and all others. Third and most important, too many researchers and practitioners in data mining do not appreciate properly the issue of statistical significance and the danger of overfitting. Given a dataset such as the one for the CoIL contest, it is pointless to apply a very complicated learning algorithm, or to perform a very time-consuming model search. In either ease, one is likely to overfit the training data and to fool oneself in estimating predictive accuracy and in discovering useful correlations.

REFERENCES

• 1: G. Bateson. The logical categories of learning and communication. In Steps to an Ecology of Mind, pages 279-308. Ballantine Books, 1972.
• 2: Thomas G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms, Neural Computation, v.10 n.7, p.1895-1923, Oct. 1998 [doi>10.1162/089976698300017197]
• 3: C. Elkan. Boosting and naive Bayesian leaming. Technical Report CS97-557, Department of Computer Science and Engineering, University of California, San Diego, 1997.
• 4: Fredrik Farnstrom , James Lewis , Charles Elkan, Scalability for clustering algorithms revisited, ACM SIGKDD Explorations Newsletter, v.2 n.1, p.51-57, June, 2000 [doi>10.1145/360402.360419]
• 5: G. Gigerenzer. Adaptive Thinking: Rationality in the Real WorM. Oxford University Press, 2000.
• 6: T. S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press, 1962.
• 7: S. D. Silvey. Statistical Inference. John Wiley & Sons, Inc., 1975. Reprinted with corrections.
• 8: CoIL challenge 2000: The insurance company case, Technical Report 2000-09, Leiden Institute of Advanced Computer Science, Netherlands, 2000. Available at www.wi, leidenuniv.nl/~putten/library/ cc2000/report .html.
• 9: R. Wirth and J. Hipp. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the Fourth International Conference on the Practical Application of Knowledge Discovery and Data Mining (PADD '00), pages 29-39, 2000.
• 10: Bianca Zadrozny , Charles Elkan, Learning and making decisions when costs and probabilities are both unknown, Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, p.204-213, August 26-29, 2001, San Francisco, California [doi>10.1145/502512.502540]
• 11: Bianca Zadrozny , Charles Elkan, Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers, Proceedings of the Eighteenth International Conference on Machine Learning, p.609-616, June 28-July 01, 2001

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Distribution functions; Correlation and regression analysis

General Terms:
Algorithms, Design, Performance