Learning from measurements in exponential families

ABSTRACT

Given a model family and a set of unlabeled examples, one could either label specific examples or state general constraints---both provide information about the desired model. In general, what is the most cost-effective way to learn? To address this question, we introduce measurements, a general class of mechanisms for providing information about a target model. We present a Bayesian decision-theoretic framework, which allows us to both integrate diverse measurements and choose new measurements to make. We use a variational inference algorithm, which exploits exponential family duality. The merits of our approach are demonstrated on two sequence labeling tasks.

REFERENCES

• 1: Borwein, J. M., & Zhu, Q. J. (2005). Techniques of variational analysis. Springer.
• 2: Chaloner, K., & Verdinelli, I. (1995). Bayesian experimental design: A review. Statistical Science, 10, 273--304.
• 3: Chang, M., Ratinov, L., & Roth, D. (2007). Guiding semi-supervision with constraint-driven learning. Association for Computational Linguistics (ACL) (pp. 280--287).
• 4: Gregory Druck , Gideon Mann , Andrew McCallum, Learning from labeled features using generalized expectation criteria, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390436]
• 5: Miroslav Dudík , Steven J. Phillips , Robert E. Schapire, Maximum Entropy Density Estimation with Generalized Regularization and an Application to Species Distribution Modeling, The Journal of Machine Learning Research, 8, p.1217-1260, 12/1/2007
• 6: Graça, J., Ganchev, K., & Taskar, B. (2008). Expectation maximization and posterior constraints. Advances in Neural Information Processing Systems (NIPS) (pp. 569--576).
• 7: Aria Haghighi , Dan Klein, Prototype-driven learning for sequence models, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.320-327, June 04-09, 2006, New York, New York [doi>10.3115/1220835.1220876]
• 8: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 9: Gideon S. Mann , Andrew McCallum, Simple, robust, scalable semi-supervised learning via expectation regularization, Proceedings of the 24th international conference on Machine learning, p.593-600, June 20-24, 2007, Corvalis, Oregon [doi>10.1145/1273496.1273571]
• 10: Mann, G., & McCallum, A. (2008). Generalized expectation criteria for semi-supervised learning of conditional random fields. Human Language Technology and Association for Computational Linguistics (HLT/ACL) (pp. 870--878).
• 11: Novi Quadrianto , Alex J. Smola , Tiberio S. Caetano , Quoc V. Le, Estimating labels from label proportions, Proceedings of the 25th international conference on Machine learning, p.776-783, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390254]
• 12: Nicholas Roy , Andrew McCallum, Toward Optimal Active Learning through Sampling Estimation of Error Reduction, Proceedings of the Eighteenth International Conference on Machine Learning, p.441-448, June 28-July 01, 2001
• 13: Matthias W. Seeger , Hannes Nickisch, Compressed sensing and Bayesian experimental design, Proceedings of the 25th international conference on Machine learning, p.912-919, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390271]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.3 Deduction and Theorem Proving
          Subjects: Inference engines
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory