Feature shaping for linear SVM classifiers

ABSTRACT

Linear classifiers have been shown to be effective for many discrimination tasks. Irrespective of the learning algorithm itself, the final classifier has a weight to multiply by each feature. This suggests that ideally each input feature should be linearly correlated with the target variable (or anti-correlated), whereas raw features may be highly non-linear. In this paper, we attempt to re-shape each input feature so that it is appropriate to use with a linear weight and to scale the different features in proportion to their predictive value. We demonstrate that this pre-processing is beneficial for linear SVM classifiers on a large benchmark of text classification tasks as well as UCI datasets.

REFERENCES

• 1: David A. Cieslak , Nitesh V. Chawla, Learning Decision Trees for Unbalanced Data, Proceedings of the 2008 European Conference on Machine Learning and Knowledge Discovery in Databases - Part I, September 15-19, 2008, Antwerp, Belgium [doi>10.1007/978-3-540-87479-9_34]
• 2: C. Cortes, A. Gretton, G. Lackriet, M. Mohri, and A. Rostamizadeh, editors. NIPS Kernel Learning Workshop. 2008. http://www.cs.nyu.edu/learning_kernels/
• 3: Franca Debole , Fabrizio Sebastiani, Supervised term weighting for automated text categorization, Proceedings of the 2003 ACM symposium on Applied computing, March 09-12, 2003, Melbourne, Florida [doi>10.1145/952532.952688]
• 4: George Forman, An extensive empirical study of feature selection metrics for text classification, The Journal of Machine Learning Research, 3, 3/1/2003
• 5: George Forman, BNS feature scaling: an improved representation over tf-idf for svm text classification, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA [doi>10.1145/1458082.1458119]
• 6: Evgeniy Gabrilovich , Shaul Markovitch, Text categorization with many redundant features: using aggressive feature selection to make SVMs competitive with C4.5, Proceedings of the twenty-first international conference on Machine learning, p.41, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015388]
• 7: Isabelle Guyon , André Elisseeff, An introduction to variable and feature selection, The Journal of Machine Learning Research, 3, 3/1/2003
• 8: Eui-Hong Han , George Karypis, Centroid-Based Document Classification: Analysis and Experimental Results, Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, p.424-431, September 13-16, 2000
• 9: Huan Liu , Hiroshi Motoda, Feature Selection for Knowledge Discovery and Data Mining, Kluwer Academic Publishers, Norwell, MA, 1998
• 10: Huan Liu , Hiroshi Motoda, Computational Methods of Feature Selection (Chapman & Hall/Crc Data Mining and Knowledge Discovery Series), Chapman & Hall/CRC, 2007
• 11: A. Knobbe, B. Cremilleux, J. Furnkranz, and M. Scholz. From local patterns to global models: The LeGo approach to data mining. In: Proc. of ECML/PKDD-08 Workshop: From Local Patterns to Global Models (LeGo'08), 2008. http://www.ke.informatik.tu-darmstadt.de/events/LeGo-08/
• 12: M. Lan, C. L. Tan, J. Su, and Y. Lu. Supervised and traditional term weighting methods for automatic text categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 99(2), 2008.
• 13: Gert R. G. Lanckriet , Nello Cristianini , Peter Bartlett , Laurent El Ghaoui , Michael I. Jordan, Learning the Kernel Matrix with Semidefinite Programming, The Journal of Machine Learning Research, 5, p.27-72, 12/1/2004
• 14: C. S. Ong and A. J. Smola. Machine learning with hyperkernels. In: Proc. of International Conference on Machine Learning (ICML'03), pages 568--575, 2003.
• 15: J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pages 61--74. MIT Press, 1999.
• 16: Marie Szafranski , Yves Grandvalet , Alain Rakotomamonjy, Composite kernel learning, Proceedings of the 25th international conference on Machine learning, p.1040-1047, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390287]
• 17: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Experimentation, Performance

Keywords:
SVM, feature scaling, feature weighting, linear support vector machine, machine learning, text classification