A PSO-based framework for dynamic SVM model selection

ABSTRACT

Support Vector Machines (SVM) are very powerful classifiers in theory but their efficiency in practice rely on an optimal selection of hyper-parameters. A naïve or ad hoc choice of values for the latter can lead to poor performance in terms of generalization error and high complexity of parameterized models obtained in terms of the number of support vectors identified. This hyper-parameter estimation with respect to the aforementioned performance measures is often called the model selection problem in the SVM research community. In this paper we propose a strategy to select optimal SVM models in a dynamic fashion in order to attend that when knowledge about the environment is updated with new observations and previously parameterized models need to be re-evaluated, and in some cases discarded in favour of revised models. This strategy combines the power of the swarm intelligence theory with the conventional grid-search method in order to progressively identify and sort out potential solutions using dynamically updated training datasets. Experimental results demonstrate that the proposed method outperforms the traditional approaches tested against it while saving considerable computational time.

REFERENCES

• 1: N. Ayat, M. Cheriet, and C. Suen. Automatic model selection for the optimization of SVM kernels. Pattern Recognition, 38(10):1733--1745, October 2005.
• 2: C. L. Blake and C. J. Merz. UCI repository of machine learning databases, 1998.
• 3: A. Carlisle and G. Dozier. Tracking changing extrema with adaptive particle swarm optimizer. In Procs of the 5th BWAC, pages 265--270, Orlando, USA, 2002.
• 4: C. C. Chang and C. J. Lin. Libsvm: a library for support vector machines, 2005.
• 5: O. Chapelle and V. Vapnik. Model selection for support vector machines. In Advances in Neural Information Processing Systems, pages 230--236, 1999.
• 6: C. Chatelain , S. Adam , Y. Lecourtier , L. Heutte , T. Paquet, Multi-Objective Optimization for SVM Model Selection, Proceedings of the Ninth International Conference on Document Analysis and Recognition, p.427-431, September 23-26, 2007
• 7: M. Clerc. Particle Swarm Optimization. ISTE Publishing Company, London, 2006.
• 8: M. Clerc and J. Kennedy. The particle swarm - explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation, 6(1):58--73, 2002.
• 9: Lior Cohen , Gil Avrahami-Bakish , Mark Last , Abraham Kandel , Oscar Kipersztok, Real-time data mining of non-stationary data streams from sensor networks, Information Fusion, v.9 n.3, p.344-353, July, 2008 [doi>10.1016/j.inffus.2005.05.005]
• 10: Bruno Feres de Souza , Andre C. P. L. F. de Carvalho , Rodrigo Calvo , Renato Porfirio Ishii, Multiclass SVM Model Selection Using Particle Swarm Optimization, Proceedings of the Sixth International Conference on Hybrid Intelligent Systems, p.31, December 13-15, 2006 [doi>10.1109/HIS.2006.50]
• 11: F. Friedrichs and C. Igel. Evolutionary tuning of multiple SVM parameters. In Proceedings of the 12th ESANN, pages 519--524, 2004.
• 12: P. Henniges, E. Granger, and R. Sabourin. Factors of overtraining with fuzzy ARTMAP neural networks. In Procs of the IJCNN, pages 1--4, 2005.
• 13: Xiaohui Hu , R. C. Eberhart, Adaptive particle swarm optimization: detection and response to dynamic systems, Proceedings of the Evolutionary Computation on 2002. CEC '02. Proceedings of the 2002 Congress, p.1666-1670, May 12-17, 2002
• 14: Ming-Hui Jiang , Xu-Chuan Yuan, Construction and Application of PSO-SVM Model for Personal Credit Scoring, Proceedings of the 7th international conference on Computational Science, Part IV: ICCS 2007, p.158-161, May 27-30, 2007, Beijing, China [doi>10.1007/978-3-540-72590-9_22]
• 15: Y. Jin and J. Branke. Evolutionary optimization in uncertain environments - A survey. IEEE Transactions on Evolutionary Computation, 9(3):303--317, 2005.
• 16: J. Kennedy. Some issues and practices for particle swarms. In Procs of IEEE SIS, pages 801--808, 2007.
• 17: J. Kennedy and R. C. Eberhart. Particle swarm intelligence. In Procs of ICNN, pages 1942--1948, 1995.
• 18: A. Nickabadi, M. M. Ebadzadeh, and R. Safabakhsh. DNPSO: A dynamic niching particle swarm optimizer for multi-modal optimization. In Procs of the IEEE CEC, pages 26--32, 2008.
• 19: Luiz S. Oliveira , Robert Sabourin , Flávio Bortolozzi , Ching Y. Suen, Automatic Recognition of Handwritten Numerical Strings: A Recognition and Verification Strategy, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.11, p.1438-1454, November 2002 [doi>10.1109/TPAMI.2002.1046154]
• 20: Y. Park , J. Sklansky, Automated design of linear tree classifiers, Pattern Recognition, v.23 n.12, p.1393-1412, 1990 [doi>10.1016/0031-3203(90)90086-Z]
• 21: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

General Terms:
Experimentation

Keywords:
dynamic optimization, model selection, particle swarm optimization, support vector machines