An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators

ABSTRACT

Statistical and computational concerns have motivated parameter estimators based on various forms of likelihood, e.g., joint, conditional, and pseudolikelihood. In this paper, we present a unified framework for studying these estimators, which allows us to compare their relative (statistical) efficiencies. Our asymptotic analysis suggests that modeling more of the data tends to reduce variance, but at the cost of being more sensitive to model misspecification. We present experiments validating our analysis.

REFERENCES

• 1: Besag, J. (1975). The analysis of non-lattice data. The Statistician, 24, 179--195.
• 2: Bouchard, G., & Triggs, B. (2004). The trade-off between generative and discriminative classifiers. International Conference on Computational Statistics (pp. 721--728).
• 3: John D. Lafferty , Andrew McCallum , Fernando C. N. Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning, p.282-289, June 28-July 01, 2001
• 4: Julia A. Lasserre , Christopher M. Bishop , Thomas P. Minka, Principled Hybrids of Generative and Discriminative Models, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.87-94, June 17-22, 2006 [doi>10.1109/CVPR.2006.227]
• 5: Liang, P., Klein, D., & Jordan, M. I. (2008). Agreement-based learning. Advances in Neural Information Processing Systems (NIPS).
• 6: Lindsay, B. (1988). Composite likelihood methods. Contemporary Mathematics, 80, 221--239.
• 7: McCallum, A., Pal, C., Druck, G., & Wang, X. (2006). Multi-conditional learning: Generative/discriminative training for clustering and classification. Association for the Advancement of Artificial Intelligence (AAAI).
• 8: Ng, A. Y., & Jordan, M. I. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. Advances in Neural Information Processing Systems (NIPS).
• 9: Sutton, C., & McCallum, A. (2005). Piecewise training of undirected models. Uncertainty in Artificial Intelligence (UAI).
• 10: van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.
• 11: Varin, C. (2008). On composite marginal likelihoods. Advances in Statistical Analysis, 92, 1--28.
• 12: Martin J. Wainwright, Estimating the "Wrong" Graphical Model: Benefits in the Computation-Limited Setting, The Journal of Machine Learning Research, 7, p.1829-1859, 12/1/2006
• 13: Wainwright, M., Jaakkola, T., & Willsky, A. (2003). Treereweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching. Artificial Intelligence and Statistics (AISTATS).
• 14: Wainwright, M., & Jordan, M. I. (2003). Graphical models, exponential families, and variational inference (Technical Report). Department of Statistics, University of California at Berkeley.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory