Using ambiguity measure feature selection algorithm for support vector machine classifier

ABSTRACT

With the ever-increasing number of documents on the web, digital libraries, news sources, etc., the need of a text classifier that can classify massive amount of data is becoming more critical and difficult. The major problem in text classification is the high dimensionality of feature space. The Support Vector Machine (SVM) classifier is shown to perform consistently better than other text classification algorithms. However, the time taken for training a SVM model is more than other algorithms. We explore the use of the Ambiguity Measure (AM) feature selection method that uses only the most unambiguous keywords to predict the category of a document. Our analysis shows that AM reduces the training time by more than 50% than the scenario when no feature selection is used, while maintaining the accuracy of the text classifier equivalent to or better than using the whole feature set. We empirically show the effectiveness of our approach in outperforming seven different feature selection methods using two standard benchmark datasets.

REFERENCES

• 1: Chang C. C., Lin C. J., LIBSVM: a library for support vector machines, 2001.
• 2: Bong Chih How , K. Narayanan, An Empirical Study of Feature Selection for Text Categorization based on Term Weightage, Proceedings of the 2004 IEEE/WIC/ACM International Conference on Web Intelligence, p.599-602, September 20-24, 2004 [doi>10.1109/WI.2004.24]
• 3: Corinna Cortes , Vladimir Vapnik, Support-Vector Networks, Machine Learning, v.20 n.3, p.273-297, Sept. 1995 [doi>10.1023/A:1022627411411]
• 4: Thorsten Joachims, Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 5: Thorsten Joachims, Text Categorization with Suport Vector Machines: Learning with Many Relevant Features, Proceedings of the 10th European Conference on Machine Learning, p.137-142, April 21-23, 1998
• 6: http://en.wikipedia.org/wiki/Metallica.
• 7: Lang K., Original 20 Newsgroups Dataset. http://people.csai.mit.edu/jrennie/20Newsgroups.
• 8: Lewis D., Reuters-21578, http://www.daviddlewis.com/resources/testcollections/reuters21578.
• 9: Mengle S., Goharian N., Platt Alana., FACT: Fast Algorithm for Categorizing Text. IEEE 5^th International Conference on Intelligence and Security Informatics, 2007. pg 308--315.
• 10: Donald Michie , D. J. Spiegelhalter , C. C. Taylor , John Campbell, Machine learning, neural and statistical classification, Ellis Horwood, Upper Saddle River, NJ, 1995
• 11: Dunja Mladenić , Janez Brank , Marko Grobelnik , Natasa Milic-Frayling, Feature selection using linear classifier weights: interaction with classification models, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009034]
• 12: Novovicova J., Malik A., Information-theoretic feature selection algorithms for text classification. IEEE International Joint Conference on Neural Networks, IJCNN 2005. Volume: 5, pg 3272--3277.
• 13: Wenqian Shang , Houkuan Huang , Haibin Zhu , Yongmin Lin , Youli Qu , Zhihai Wang, A novel feature selection algorithm for text categorization, Expert Systems with Applications: An International Journal, v.33 n.1, p.1-5, July, 2007 [doi>10.1016/j.eswa.2006.04.001]
• 14: Jun Yan , Ning Liu , Benyu Zhang , Shuicheng Yan , Zheng Chen , Qiansheng Cheng , Weiguo Fan , Wei-Ying Ma, OCFS: optimal orthogonal centroid feature selection for text categorization, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076058]
• 15: Yiming Yang , Jan O. Pedersen, A Comparative Study on Feature Selection in Text Categorization, Proceedings of the Fourteenth International Conference on Machine Learning, p.412-420, July 08-12, 1997
• 16: Yiming Yang , Jian Zhang , Bryan Kisiel, A scalability analysis of classifiers in text categorization, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860455]
• 17: Zheng Z., Srihari R., Optimally Combining Positive and Negative Features for Text Categorization. In Proceedings of the ICML, Workshop on Learning from Imbalanced Datasets II, Washington DC, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Performance

Keywords:
SVM, feature selection, text classification