Partially supervised feature selection with regularized linear models

ABSTRACT

This paper addresses feature selection techniques for classification of high dimensional data, such as those produced by microarray experiments. Some prior knowledge may be available in this context to bias the selection towards some dimensions (genes) a priori assumed to be more relevant. We propose a feature selection method making use of this partial supervision. It extends previous works on embedded feature selection with linear models including regularization to enforce sparsity. A practical approximation of this technique reduces to standard SVM learning with iterative rescaling of the inputs. The scaling factors depend here on the prior knowledge but the final selection may depart from it. Practical results on several microarray data sets show the benefits of the proposed approach in terms of the stability of the selected gene lists with improved classification performances.

REFERENCES

• 1: Alon, U., Barkai, N., Notterman, D., Gish, K., Ybarra, S., Mack, D., & Levine, A. (1999). Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. PNAS, 96, 6745--6750.
• 2: Ambroise, C., & McLachlan, G. (2002). Selection bias in gene extraction on the basis of microarra gene-expression data. PNAS, 99, 6562--6566.
• 3: Olivier Chapelle , Vladimir Vapnik , Olivier Bousquet , Sayan Mukherjee, Choosing Multiple Parameters for Support Vector Machines, Machine Learning, v.46 n.1-3, p.131-159, 2002 [doi>10.1023/A:1012450327387]
• 4: Cheng, Y., Cai, Y., Sun, Y., & Li, J. (2008). Semi-supervised feature selection under logistic I-RELIEF framework. 19th International Conference on Pattern Recognition.
• 5: Franke, M., & Wolfe, P. (1956). An algorithm for quadratic programming. Naval Research Logistics Quaterly, 3, 95--110.
• 6: Golub, T., Slonim, D., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J., Coller, H., Loh, M., Downing, J., Caligiuri, M., Bloomfield, C., & Lander, E. (1999). Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science, 286, 531--537.
• 7: Isabelle Guyon , André Elisseeff, An introduction to variable and feature selection, The Journal of Machine Learning Research, 3, 3/1/2003
• 8: Isabelle Guyon , Jason Weston , Stephen Barnhill , Vladimir Vapnik, Gene Selection for Cancer Classification using Support Vector Machines, Machine Learning, v.46 n.1-3, p.389-422, 2002 [doi>10.1023/A:1012487302797]
• 9: Krishnapuram, B., Carin, L., & Hartemink, A. (2004). Kernel methods in computational biology, chapter 14: Gene Expression Analysis: Joint Feature Selection and Classifier Design, 299--317. Cambridge, MA: MIT Press.
• 10: Ludmila I. Kuncheva, A stability index for feature selection, Proceedings of the 25th conference on Proceedings of the 25th IASTED International Multi-Conference: artificial intelligence and applications, p.390-395, February 12-14, 2007, Innsbruck, Austria
• 11: Mukherjee, S. (2003). A practical approach to microarray data analysis, chapter 9: Classifying Microarray Data Using Support Vector Machines, 166--185. Springer.
• 12: Roth, V. (2004). The generalized LASSO. IEEE Transactions on Neural Networks, 15, 16--28.
• 13: Yvan Saeys , Iñaki Inza , Pedro Larrañaga, A review of feature selection techniques in bioinformatics, Bioinformatics, v.23 n.19, p.2507-2517, September 2007 [doi>10.1093/bioinformatics/btm344]
• 14: Shipp, M. A., Ross, K. N., Tamayo, P., Weng, A. P., Kutok, J. L., Aguiar, R. C., Gaasenbeek, M., Angelo, M., Reich, M., Pinkus, G. S., Ray, T. S., Koval, M. A., Last, K. W., Norton, A., Lister, T. A., Mesirov, J., Neuberg, D. S., Lander, E. S., Aster, J. C., & Golub, T. R. (2002). Diffuse large b-cell lymphoma outcome prediction by gene-expression profiling and supervised machine learning. Nature Medicine, 8, 68--74.
• 15: Singh, D., Febbo, P., Ross, K., Jackson, D., Manola, J., Ladd, C., Tamayo, P., Renshaw, A., D'Amico, A., Richie, J., Lander, E., Loda, M., Kantoff, P., Golub, T., & Sellers, W. (2002). Gene expression correlates of clinical prostate cancer behavior. Cancer Cell, 1, 203--209.
• 16: Li Wang , Ji Zhu , Hui Zou, Hybrid huberized support vector machines for microarray classification, Proceedings of the 24th international conference on Machine learning, p.983-990, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273620]
• 17: Jason Weston , André Elisseeff , Bernhard Schölkopf , Mike Tipping, Use of the zero norm with linear models and kernel methods, The Journal of Machine Learning Research, 3, 3/1/2003
• 18: Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., & Vapnik, V. (2000). Feature selection for SVMs. Advances in Neural Information Processing Systems (pp. 668--674).
• 19: Zhao, Z., & Liu, H. (2007). Semi-supervised feature selection via spectral analysis. 7th SIAM International Conference on Data Mining (pp. 641--652).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES

General Terms:
Algorithms, Design, Performance, Theory