Gaussian process classification for segmenting and annotating sequences

ABSTRACT

Many real-world classification tasks involve the prediction of multiple, inter-dependent class labels. A prototypical case of this sort deals with prediction of a sequence of labels for a sequence of observations. Such problems arise naturally in the context of annotating and segmenting observation sequences. This paper generalizes Gaussian Process classification to predict multiple labels by taking dependencies between neighboring labels into account. Our approach is motivated by the desire to retain rigorous probabilistic semantics, while overcoming limitations of parametric methods like Conditional Random Fields, which exhibit conceptual and computational difficulties in high-dimensional input spaces. Experiments on named entity recognition and pitch accent prediction tasks demonstrate the competitiveness of our approach.

REFERENCES

• 1: Altun, Y., Hofmann, T., & Johnson, M. (2003a). Discriminative learning for label sequences via boosting. Advances in Neural Information Processing Systems.
• 2: Altun, Y., Tsochantaridis, I., & Hofmann, T. (2003b). Hidden markov support vector machines. 20th International Conference on Machine Learning.
• 3: Kristin P. Bennett , Michinari Momma , Mark J. Embrechts, MARK: a boosting algorithm for heterogeneous kernel models, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775051]
• 4: Michael Collins, Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.1-8, July 06, 2002 [doi>10.3115/1118693.1118694]
• 5: Koby Crammer , Yoram Singer, On the algorithmic implementation of multiclass kernel-based vector machines, The Journal of Machine Learning Research, 2, 3/1/2002
• 6: Lehel Csató , Manfred Opper, Sparse on-line Gaussian processes, Neural Computation, v.14 n.3, p.641-668, March 2002 [doi>10.1162/089976602317250933]
• 7: Gibbs, M. N., & MacKay, D. J. C. (2000). Variational Gaussian Process Classifiers. IEEE-NN, 11, 1458.
• 8: Federico Girosi, An Equivalence Between Sparse Approximation and Support Vector Machines, Massachusetts Institute of Technology, Cambridge, MA, 1997
• 9: Greenberg, S., Ellis, D., & Hollenback, J. (1996). Insights into spoken language gleaned from phonetic transcripti on of the Switchboard corpus. ICSLP96.
• 10: Jaakkola, T. S., & Jordan, M. I. (1996). Computing upper and lower bounds on likelihoods in intractable networks. In Proc. of the Twelfth Conf. on UAI.
• 11: Kimeldorf, G., & Wahba, G. (1971). A correspondence between Bayesian estimation and on stochastic processes and smoothing by splines. Annals of Math. Stat., 41(2), 495--502.
• 12: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 13: Andrew McCallum , Dayne Freitag , Fernando C. N. Pereira, Maximum Entropy Markov Models for Information Extraction and Segmentation, Proceedings of the Seventeenth International Conference on Machine Learning, p.591-598, June 29-July 02, 2000
• 14: Thomas P. Minka , Rosalind Picard, A family of algorithms for approximate bayesian inference, 2001
• 15: Manfred Opper , Ole Winther, Gaussian Processes for Classification: Mean-Field Algorithms, Neural Computation, v.12 n.11, p.2655-2684, November 2000 [doi>10.1162/089976600300014881]
• 16: Punyakanok, V., & Roth, D. (2000). The use of classifiers in sequential inference. Advances in Neural Information Processing Systems.
• 17: Schölkopf, B., & Smola, A. J. (2002). Learning with kernels. MIT Press.
• 18: Seeger, M., Lawrence, N. D., & Herbrich, R. (2003). Fast sparse Gaussian Process methods: The informative vector machine. Advances in Neural Information Processing Systems.
• 19: Smola, A. J., & Bartlett, P. L. (2000). Sparse greedy Gaussian Process regression. Advances in Neural Information Processing Systems.
• 20: Alex J. Smola , Bernhard Schökopf, Sparse Greedy Matrix Approximation for Machine Learning, Proceedings of the Seventeenth International Conference on Machine Learning, p.911-918, June 29-July 02, 2000
• 21: Taskar, B., Guestrin, C., & Koller, D. (2004). Max-margin markov networks. Advances in Neural Information Processing Systems.
• 22: Weston, J., & Watkins, C. (1999). Support vector machines for multi-class pattern recognition. Proceedings European Symposium on Artificial Neural Networks.
• 23: Christopher K. I. Williams , David Barber, Bayesian Classification With Gaussian Processes, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.12, p.1342-1351, December 1998 [doi>10.1109/34.735807]
• 24: Williams, C. K. I., & Seeger, M. (2000). Using the nystrom method to speed up kernel machines. Advances in Neural Information Processing Systems.
• 25: Zhu, & Hastie, T. (2001). Kernel logistic regression and the import vector machine. Advances in Neural Information Processing Systems.