Support vector regression for classifier prediction

ABSTRACT

In this paper we introduce XCSF with support vector prediction:the problem of learning the prediction function is solved as a support vector regression problem and each classifier exploits a Support Vector Machine to compute the prediction. In XCSF with support vector prediction, XCSFsvm, the genetic algorithm adapts classifier conditions, classifier actions, and the SVM kernel parameters.We compare XCSF with support vector prediction to XCSF with linear prediction on the approximation of four test functions.Our results suggest that XCSF with support vector prediction compared to XCSF with linear prediction (i) is able to evolve accurate approximations of more difficult functions, (ii) has better generalization capabilities and (iii) learns faster.

REFERENCES

• 1: Martin V. Butz, Kernel-based, ellipsoidal conditions in the real-valued XCS classifier system, Proceedings of the 2005 conference on Genetic and evolutionary computation, June 25-29, 2005, Washington DC, USA [doi>10.1145/1068009.1068320]
• 2: Martin V. Butz, Pier-Luca Lanzi, and Stewart W. Wilson. Function approximation with XCS: Hyperellipsoidal conditions, recursive least squares, and compaction. (Available on request from www.illigal.org/butz), 2007.
• 3: Martin V. Butz and Stewart W. Wilson. An algorithmic description of XCS. Journal of Soft Computing, 6(3---4):144---153, 2002.
• 4: Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. In NIPS, pages 409--415, 2000.
• 5: Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/$\backsim$cjlin/libsvm, 2001.
• 6: Ronan Collobert , Samy Bengio, SVMTorch: support vector machines for large-scale regression problems, The Journal of Machine Learning Research, 1, p.143-160, 9/1/2001
• 7: Yaakov Engel , Shie Mannor , Ron Meir, Sparse Online Greedy Support Vector Regression, Proceedings of the 13th European Conference on Machine Learning, p.84-96, August 19-23, 2002
• 8: S. A. Glantz and B. K. Slinker. Primer of Applied Regression & Analysis of Variance. McGraw Hill, 2001. second edition.
• 9: Pier Luca Lanzi, Daniele Loiacono, Stewart W. Wilson, and Dave E. Goldberg. Generalization in the XCSF classifier system: Analysis, improvement, and extension. Technical Report 2005012, Illinois Genetic Algorithms Laboratory -- University of Illinois at Urbana-Champaign, 2005.
• 10: Pier Luca Lanzi , Daniele Loiacono , Stewart W. Wilson , David E. Goldberg, Extending XCSF beyond linear approximation, Proceedings of the 2005 conference on Genetic and evolutionary computation, June 25-29, 2005, Washington DC, USA [doi>10.1145/1068009.1068319]
• 11: Pier Luca Lanzi, Daniele Loiacono, Stewart~W. Wilson, and David E. Goldberg. XCS with computed prediction for the learning of boolean functions. In Proceedings of the IEEE Congress on Evolutionary Computation -- CEC--2005, pages 588--595, Edinburgh, UK, September 2005. IEEE.
• 12: Pier Luca Lanzi , Daniele Loiacono , Stewart W. Wilson , David E. Goldberg, Prediction update algorithms for XCSF: RLS, Kalman filter, and gain adaptation, Proceedings of the 8th annual conference on Genetic and evolutionary computation, July 08-12, 2006, Seattle, Washington, USA [doi>10.1145/1143997.1144243]
• 13: Pavel Laskov , Christian Gehl , Stefan Krüger , Klaus-Robert Müller, Incremental Support Vector Learning: Analysis, Implementation and Applications, The Journal of Machine Learning Research, 7, p.1909-1936, 12/1/2006
• 14: Junshui Ma , James Theiler , Simon Perkins, Accurate on-line support vector regression, Neural Computation, v.15 n.11, p.2683-2703, November 2003 [doi>10.1162/089976603322385117]
• 15: Mario Martin. On-line support vector machines for function approximation. Technical report, Universitat Politecnica de Catalunya, 2002.
• 16: John C. Platt, Fast training of support vector machines using sequential minimal optimization, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 17: Liva Ralaivola , Florence d'Alché-Buc, Incremental Support Vector Machine Learning: A Local Approach, Proceedings of the International Conference on Artificial Neural Networks, p.322-330, August 21-25, 2001
• 18: R. Rosipal and M. Gorilami. An adaptive support vector regression filter: A signal detection application. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 2, pages 603--607 vol.2, 7-10 Sept. 1999.
• 19: Alex J. Smola , Bernhard Schölkopf, A tutorial on support vector regression, Statistics and Computing, v.14 n.3, p.199-222, August 2004 [doi>10.1023/B:STCO.0000035301.49549.88]
• 20: Nadeem Ahmed Syed , Huan Liu , Kah Kay Sung, Handling concept drifts in incremental learning with support vector machines, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.317-321, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312267]
• 21: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 22: S. Vijayakumar and S. Wu. Sequential support vector classifiers and regression. In Int. Conf. Soft Computing}, pages 610--619, 1999.
• 23: Bernard Widrow , Marcian E. Hoff, Adaptive switching circuits, Neurocomputing: foundations of research, MIT Press, Cambridge, MA, 1988
• 24: Stewart W. Wilson. Classifier Fitness Based on Accuracy. Evolutionary Computation, 3(2):149--175, 1995. http://prediction-dynamics.com/.
• 25: Stewart W. Wilson, Mining Oblique Data with XCS, Revised Papers from the Third International Workshop on Advances in Learning Classifier Systems, p.158-176, September 15-16, 2000
• 26: Stewart W. Wilson, Classifiers that approximate functions, Natural Computing: an international journal, v.1 n.2-3, p.211-234, June 2002 [doi>10.1023/A:1016535925043]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

General Terms:
Algorithms, Performance

Keywords:
XCS, computed prediction, genetic algorithms, learning classifier systems, support vector machines