SVM binary classifier ensembles for image classification

ABSTRACT

We study how the SVM-based binary classifiers can be effectively combined to tackle the multi-class image classification problem. We study several ensemble schemes, including OPC (one per class), PWC (pairwise coupling), and ECOC (error-correction output coding), that aim to achieve good error correction capability through redundancy. To enhance these ensemble schemes' accuracy, we propose methods that on the one hand boost the margins (i.e., confidence) of the SVM-based binary classifiers, and, on the other hand, remove the noise of irrelevant classifiers from class prediction. From empirical study we show that our margin boosting and noise reduction methods lead to higher classification accuracy than ensemble schemes that are solely designed for maximum error correction capability.

REFERENCES

• 1: Erin L. Allwein , Robert E. Schapire , Yoram Singer, Reducing multiclass to binary: a unifying approach for margin classifiers, The Journal of Machine Learning Research, 1, p.113-141, 9/1/2001 [doi>10.1162/15324430152733133]
• 2: Leo Breiman, Bagging predictors, Machine Learning, v.24 n.2, p.123-140, Aug. 1996 [doi>10.1023/A:1018054314350]
• 3: L. Breiman. Arcing classifiers. The Annals of Statistics, pages 801-849, 1997.
• 4: Christopher J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, v.2 n.2, p.121-167, June 1998 [doi>10.1023/A:1009715923555]
• 5: Edward Y. Chang , Beitao Li , Chen Li, Toward Perception-Based Image Retrieval, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'00), p.101, June 16-16, 2000
• 6: T. Dietterich and G. Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of Artijkal Intelligence Research, 2, 1995.
• 7: J. Friedman. Another approach to polychotomous classification. Stanford University Technical Report, 1996.
• 8: Adam J. Grove , Dale Schuurmans, Boosting in the limit: maximizing the margin of learned ensembles, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.692-699, July 1998, Madison, Wisconsin, United States
• 9: G. James. Variance and bias for general loss functions. -USC Technical Report! 2001.
• 10: G. James and T. Hastie. Error Coding and PaCT's. 1997.
• 11: Thorsten Joachims, Transductive Inference for Text Classification using Support Vector Machines, Proceedings of the Sixteenth International Conference on Machine Learning, p.200-209, June 27-30, 1999
• 12: B. Li, E. Chang, and C.-S. Li. Learning image query concepts via intelligent sampling. Proceedings of IEEE Multimedia, August 2001.
• 13: Beitao Li , Wei-Cheng Lai , Edward Y. Chang , Kwang-Ting Cheng, Mining Image Features for Efficient Query Processing, Proceedings of the 2001 IEEE International Conference on Data Mining, p.353-360, November 29-December 02, 2001
• 14: Miguel Moreira , Eddy Mayoraz, Improved Pairwise Coupling Classification with Correcting Classifiers, Proceedings of the 10th European Conference on Machine Learning, p.160-171, April 21-23, 1998
• 15: J. Platt. Probabilistic outputs for svms and comparisons to regularized likelihood methods. In Advances in Large Mar-gin Classijiers. MIT Press, 1999.
• 16: J. Platt, N. Cristianini, and J. Shawe-Taylor. Large margin dags for multiclass classification. In Advances in Neural Information Processing System 1.9, pages 547-553. MIT Press, 2000.
• 17: Robert E. Schapire , Yoav Freund , Peter Barlett , Wee Sun Lee, Boosting the margin: A new explanation for the effectiveness of voting methods, Proceedings of the Fourteenth International Conference on Machine Learning, p.322-330, July 08-12, 1997
• 18: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 19: G. Valentini. Upper bounds on the training error of ecoc svm ensembles. Technical Report, Feb. 2001.
• 20: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995
• 21: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.0 General
          Subjects: Error analysis

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
          Subjects: Feature representation

General Terms:
Algorithms, Design, Performance