Feature selection using linear classifier weights: interaction with classification models

ABSTRACT

This paper explores feature scoring and selection based on weights from linear classification models. It investigates how these methods combine with various learning models. Our comparative analysis includes three learning algorithms: Naïve Bayes, Perceptron, and Support Vector Machines (SVM) in combination with three feature weighting methods: Odds Ratio, Information Gain, and weights from linear models, the linear SVM and Perceptron. Experiments show that feature selection using weights from linear SVMs yields better classification performance than other feature weighting methods when combined with the three explored learning algorithms. The results support the conjecture that it is the sophistication of the feature weighting method rather than its apparent compatibility with the learning algorithm that improves classification performance.

REFERENCES

• 1: Janez Brank, Marko Grobelnik, Nataša Milić-Frayling, and Dunja Mladenić. Feature selection using support vector machines. Proc. of the 3rd Int. Conf. on Data Mining Methods and Databases for Engineering, Finance, and Other Fields, Bologna, Italy, September 2002.
• 2: Corinna Cortes , Vladimir Vapnik, Support-Vector Networks, Machine Learning, v.20 n.3, p.273-297, Sept. 1995 [doi>10.1023/A:1022627411411]
• 3: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 4: Werner Krauth and Marc Mézard. Learning algorithms with optimal stability in neural networks. Jour. Physics A 20, L745-L752, August 1987.
• 5: Andrew McCallum and Kamal Nigam. A comparison of event models for Naïve Bayes text categorization. AAAI Workshop on Learning for Text Categorization (pp. 41--48). AAAI Press, 1998.
• 6: Dunja Mladenic , Marko Grobelnik, Feature Selection for Unbalanced Class Distribution and Naive Bayes, Proceedings of the Sixteenth International Conference on Machine Learning, p.258-267, June 27-30, 1999
• 7: J. Ross Quinlan. Constructing decision trees. In: C4.5: Programs for machine learning, pp. 17--26. Morgan Kaufmann, 1993.
• 8: F. Rosenblatt, The perception: a probabilistic model for information storage and organization in the brain, Neurocomputing: foundations of research, MIT Press, Cambridge, MA, 1988
• 9: Vikas Sindhwani, Pushpak Bhattacharya, and Subrata Rakshit. Information theoretic feature crediting in multiclass Support Vector Machines. 1st SIAM Int. Conf. on Data Mining (SDM 2001), Chicago, IL, USA, April 5-7, 2001. SIAM, 2001.
• 10: Lawrence Shih, Yu-Han Chang, Jason Rennie, David Karger. Not too hot, not too cold: The Bundled-SVM is just right! Workshop on Text Learning (TextML-2002), ICML, Sydney, Australia, July 8, 2002.
• 11: Soumen Chakrabarti, Shourya Roy, Mahesh V. Soundalgekar: Fast and accurate text classification via multiple linear discriminant projections. Proceedings of the 28th International Conference on Very Large Data Bases (VLDB 2002), Hong Kong, China, August 20--23, 2002, pp. 658--669.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

General Terms:
Algorithms, Experimentation, Performance

Keywords:
SVM normal, feature scoring, feature selection, information retrieval, linear SVM, text classification, vector representation