Learning to learn with the informative vector machine

ABSTRACT

This paper describes an efficient method for learning the parameters of a Gaussian process (GP). The parameters are learned from multiple tasks which are assumed to have been drawn independently from the same GP prior. An efficient algorithm is obtained by extending the informative vector machine (IVM) algorithm to handle the multi-task learning case. The multi-task IVM (MTIVM) saves computation by greedily selecting the most informative examples from the separate tasks. The MT-IVM is also shown to be more efficient than random sub-sampling on an artificial data-set and more effective than the traditional IVM in a speaker dependent phoneme recognition task.

REFERENCES

• 1: Jonathan Baxter, Learning internal representations, Proceedings of the eighth annual conference on Computational learning theory, p.311-320, July 05-08, 1995, Santa Cruz, California, United States [doi>10.1145/225298.225336]
• 2: Blake, C. L., & Merz, C. J. (1998). UCI repository of machine learning databases.
• 3: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734]
• 4: Csató, L. (2002). Gaussian processes --- iterative sparse approximations. Doctoral dissertation, Aston University.
• 5: Tommi Jaakkola , Mark Diekhans , David Haussler, Using the Fisher Kernel Method to Detect Remote Protein Homologies, Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology, p.149-158, August 06-10, 1999
• 6: Lawrence, N. D., Seeger, M., & Herbrich, R. (2003). Fast sparse Gaussian process methods: The informative vector machine. Advances in Neural Information Processing Systems (pp. 625--632). Cambridge, MA: MIT Press.
• 7: Minka, T. P., & Picard, R. W. (1997). Learning how to learn is learning with point sets. Web. Revised 1999, available at http://www.stat.cmu.edu/~minka/.
• 8: Seeger, M. (2002). Covariance kernels from bayesian generative models. Advances in Neural Information Processing Systems (pp. 905--912). Cambridge, MA: MIT Press.
• 9: Thrun, S. (1996). Is learning the n-th thing any easier than learning the first? In (Touretzky et al., 1996), 640--646.
• 10: Touretzky, D. S., Mozer, M. C., & Hasselmo, M. E. (Eds.). (1996). Advances in neural information processing systems, vol. 8. Cambridge, MA: MIT Press.
• 11: C. K. I. Williams, Prediction with Gaussian processes: from linear regression to linear prediction and beyond, Proceedings of the NATO Advanced Study Institute on Learning in graphical models, p.599-621, March 1998, Erice, Italy
• 12: Williams, C. K. I., & Rasmussen, C. E. (1996). Gaussian processes for regression. In (Touretzky et al., 1996), 514--520.