Self-taught learning: transfer learning from unlabeled data

ABSTRACT

We present a new machine learning framework called "self-taught learning" for using unlabeled data in supervised classification tasks. We do not assume that the unlabeled data follows the same class labels or generative distribution as the labeled data. Thus, we would like to use a large number of unlabeled images (or audio samples, or text documents) randomly downloaded from the Internet to improve performance on a given image (or audio, or text) classification task. Such unlabeled data is significantly easier to obtain than in typical semi-supervised or transfer learning settings, making self-taught learning widely applicable to many practical learning problems. We describe an approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data. These features form a succinct input representation and significantly improve classification performance. When using an SVM for classification, we further show how a Fisher kernel can be learned for this representation.

REFERENCES

• 1: Rie Kubota Ando , Tong Zhang, A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data, The Journal of Machine Learning Research, 6, p.1817-1853, 12/1/2005
• 2: Jonathan Baxter, Theoretical models of learning to learn, Learning to learn, Kluwer Academic Publishers, Norwell, MA, 1998
• 3: Blei, D., Ng, A. Y., & Jordan, M. (2002). Latent dirichlet allocation. NIPS.
• 4: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 5: Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A. (1990). Indexing by latent semantic analysis. J. Am. Soc. Info. Sci., 41, 391--407.
• 6: Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. Ann. Stat., 32, 407--499.
• 7: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 12, p.178, June 27-July 02, 2004
• 8: Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.
• 9: Alex D. Holub , Max Welling , Pietro Perona, Combining Generative Models and Fisher Kernels for Object Recognition, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.136-143, October 17-20, 2005 [doi>10.1109/ICCV.2005.56]
• 10: Patrik O. Hoyer, Non-negative Matrix Factorization with Sparseness Constraints, The Journal of Machine Learning Research, 5, p.1457-1469, 12/1/2004
• 11: Tommi S. Jaakkola , David Haussler, Exploiting generative models in discriminative classifiers, Proceedings of the 1998 conference on Advances in neural information processing systems II, p.487-493, July 1999
• 12: Svetlana Lazebnik , Cordelia Schmid , Jean Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2169-2178, June 17-22, 2006 [doi>10.1109/CVPR.2006.68]
• 13: Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007). Efficient sparse coding algorithms. NIPS.
• 14: Andrew Y. Ng, Feature selection, L₁ vs. L₂ regularization, and rotational invariance, Proceedings of the twenty-first international conference on Machine learning, p.78, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015435]
• 15: Kamal Nigam , Andrew Kachites McCallum , Sebastian Thrun , Tom Mitchell, Text Classification from Labeled and Unlabeled Documents using EM, Machine Learning, v.39 n.2-3, p.103-134, May-June 2000 [doi>10.1023/A:1007692713085]
• 16: Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607--609.
• 17: Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290.
• 18: Thomas Serre , Lior Wolf , Tomaso Poggio, Object Recognition with Features Inspired by Visual Cortex, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.994-1000, June 20-26, 2005 [doi>10.1109/CVPR.2005.254]
• 19: Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319--2323.
• 20: Thrun, S. (1996). Is learning the n-th thing any easier than learning the first? NIPS.
• 21: Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B., 58, 267--288.
• 22: Tsuda, K., Kin, T., & Asai, K. (2002). Marginalized kernels for biological sequences. Bioinformatics, 18.
• 23: Hao Zhang , Alexander C. Berg , Michael Maire , Jitendra Malik, SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2126-2136, June 17-22, 2006 [doi>10.1109/CVPR.2006.301]