Supervised learning from multiple experts: whom to trust when everyone lies a bit

ABSTRACT

We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.

REFERENCES

• 1: Dawid, A. P., & Skeene, A. M. (1979). Maximum likei-hood estimation of observed error-rates using the EM algorithm. Applied Statistics, 28, 20--28.
• 2: Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39, 1--38.
• 3: Frank, E., & Hall, M. (2001). A simple approach to ordinal classification. Lecture Notes in Computer Science, 145--156.
• 4: Hui, S. L., & Zhou, X. H. (1998). Evaluation of diagnostic tests without a gold standard. Statistical Methods in Medical Research, 7, 354--370.
• 5: Gábor Lugosi, Learning with an unreliable teacher, Pattern Recognition, v.25 n.1, p.79-87, Jan. 1992 [doi>10.1016/0031-3203(92)90008-7]
• 6: Radford M. Neal , Geoffrey E. Hinton, A view of the EM algorithm that justifies incremental, sparse, and other variants, Proceedings of the NATO Advanced Study Institute on Learning in graphical models, p.355-368, March 1998, Erice, Italy
• 7: Victor S. Sheng , Foster Provost , Panagiotis G. Ipeirotis, Get another label? improving data quality and data mining using multiple, noisy labelers, Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2008, Las Vegas, Nevada, USA [doi>10.1145/1401890.1401965]
• 8: Smyth, P. (1995). Learning with probabilistic supervision. In Computational learning theory and natural learning systems 3, 163--182. MIT Press.
• 9: Smyth, P., Fayyad, U., Burl, M., Perona, P., & Baldi, P. (1995). Inferring ground truth from subjective labelling of venus images. In Advances in neural information processing systems 7, 1085--1092.
• 10: Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254--263).
• 11: Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. Proceedings of the First IEEE Workshop on Internet Vision at CVPR 08 (pp. 1--8).

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Probabilistic algorithms (including Monte Carlo); Experimental design

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.4 Model Validation and Analysis
      I.6.5 Model Development
          Subjects: Modeling methodologies

General Terms:
Algorithms, Design, Performance, Theory