Solving regression problems with rule-based ensemble classifiers

ABSTRACT

We describe a lightweight learning method that induces an ensemble of decision-rule solutions for regression problems. Instead of direct prediction of a continuous output variable, the method discretizes the variable by k-means clustering and solves the resultant classification problem. Predictions on new examples are made by averaging the mean values of classes with votes that are close in number to the most likely class. We provide experimental evidence that this indirect approach can often yield strong results for many applications, generally outperforming direct approaches such as regression trees and rivaling bagged regression trees.

REFERENCES

• 1: Eric Bauer , Ron Kohavi, An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants, Machine Learning, v.36 n.1-2, p.105-139, July-August 1999 [doi>10.1023/A:1007515423169]
• 2: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350]
• 3: L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth, Monterrey, CA., 1984.
• 4: R. Camacho. Inducing Models of Human Control Skills using Machine Learning Algorithms. Ph.d. thesis, University of Porto, 2000.
• 5: William W. Cohen , Yoram Singer, A simple, fast, and effective rule learner, Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, p.335-342, July 18-22, 1999, Orlando, Florida, United States
• 6: J. Dougherty, R. Kohavi, and M. Sahami. Supervised and unsupervised discretization of continuous features. In Proceedings of the 12th Int'l Conference on Machine Learning, pages 194-202, 1995.
• 7: U. Fayyad and K. Irani. Multi-interval discretization of continuous-valued attributes for classification learning. In Proceedings of the 13th Int'l Joint Conference on Artificial Intelligence, pages 1022-1027, 1993.
• 8: J. Friedman. Multivariate adaptive regression splines. Annals of Statistics, 19(1):1-141, 1991.
• 9: J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: A statistical view of boosting. Technical report, Stanford University Statistics Department, 1998. www.stat-stanford.edu/~tibs.
• 10: J. Hartigan and M. Wong. A k-means clustering algorithm, ALGORITHM AS 136. Applied Statistics, 28(1), 1979.
• 11: Robert E. Schapire , Yoav Freund , Peter Barlett , Wee Sun Lee, Boosting the margin: A new explanation for the effectiveness of voting methods, Proceedings of the Fourteenth International Conference on Machine Learning, p.322-330, July 08-12, 1997
• 12: L. Torgo and J. Gama. Regression using classification algorithms. Intelligent Data Analysis, 1(4), 1997.
• 13: S. Weiss and N. Indurkhya. Rule-based machine learning methods for functional prediction. Journal of Artificial Intelligence Research, 3:383-403, 1995.
• 14: Sholom M. Weiss , Nitin Indurkhya, Lightweight Rule Induction, Proceedings of the Seventeenth International Conference on Machine Learning, p.1135-1142, June 29-July 02, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Rule-based databases

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Robust regression

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

General Terms:
Algorithms, Performance, Theory