Detecting statistical interactions with additive groves of trees

ABSTRACT

Discovering additive structure is an important step towards understanding a complex multi-dimensional function because it allows the function to be expressed as the sum of lower-dimensional components. When variables interact, however, their effects are not additive and must be modeled and interpreted simultaneously. We present a new approach for the problem of interaction detection. Our method is based on comparing the performance of unrestricted and restricted prediction models, where restricted models are prevented from modeling an interaction in question. We show that an additive model-based regression ensemble, Additive Groves, can be restricted appropriately for use with this framework, and thus has the right properties for accurately detecting variable interactions.

REFERENCES

• 1: Rui Camacho, Inducing Models of human Control Skills, Proceedings of the 10th European Conference on Machine Learning, p.107-118, April 21-23, 1998
• 2: Rich Caruana , Mohamed Elhawary , Art Munson , Mirek Riedewald , Daria Sorokina , Daniel Fink , Wesley M. Hochachka , Steve Kelling, Mining citizen science data to predict orevalence of wild bird species, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA [doi>10.1145/1150402.1150527]
• 3: R. A. Christensen, Plane answers to complex questions: the theory of linear models, Springer-Verlag New York, Inc., New York, NY, 1987
• 4: Friedman, J. (2005). RuleFit with R. http://www-stat.stanford.edu/~jhf/R-RuleFit.html.
• 5: Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29, 1189--1232.
• 6: Friedman, J. H., & Popescu, B. E. (2005). Predictive learning via rule ensembles (Technical Report). Stanford University.
• 7: Isabelle Guyon , André Elisseeff, An introduction to variable and feature selection, The Journal of Machine Learning Research, 3, 3/1/2003
• 8: Giles Hooker, Discovering additive structure in black box functions, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014122]
• 9: Hooker, G. (2007). Generalized functional ANOVA diagnostics for high dimensional functions of dependent variables. JCGS.
• 10: Aleks Jakulin , Ivan Bratko, Testing the significance of attribute interactions, Proceedings of the twenty-first international conference on Machine learning, p.52, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015377]
• 11: Pace, R. K., & Barry, R. (1997). Sparse spatial autoregressions. Statistics and Probability Letters, 33.
• 12: Rasmussen, C. E., Neal, R. M., Hinton, G., van Camp, D., Revow, M., Ghahramani, Z., Kustra, R., & Tibshirani, R. (2003). Delve. University of Toronto. http://www.cs.toronto.edu/~delve.
• 13: Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). Semiparametric regression. Cambridge.
• 14: Daria Sorokina , Rich Caruana , Mirek Riedewald, Additive Groves of Regression Trees, Proceedings of the 18th European conference on Machine Learning, September 17-21, 2007, Warsaw, Poland [doi>10.1007/978-3-540-74958-5_31]
• 15: Torgo, L. (2007). Regression DataSets. www.liacc.up.pt/~ltorgo/Regression/DataSets.html.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis

General Terms:
Algorithms, Design, Performance