Collective revelation: a mechanism for self-verified, weighted, and truthful predictions

ABSTRACT

Decision makers can benefit from the subjective judgment of experts. For example, estimates of disease prevalence are quite valuable, yet can be difficult to measure objectively. Useful features of mechanisms for aggregating expert opinions include the ability to: (1) incentivize participants to be truthful; (2) adjust for the fact that some experts are better informed than others; and (3) circumvent the need for objective, "ground truth" observations. Subsets of these properties are attainable by previous elicitation methods, including proper scoring rules, prediction markets, and the Bayesian truth serum. Our mechanism of collective revelation, however, is the first to simultaneously achieve all three. Furthermore, we introduce a general technique for constructing budget-balanced mechanisms-where no net payments are made to participants--that applies both to collective revelation and to past peer-prediction methods.

REFERENCES

• 1: G. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1--3, 1950.
• 2: Yiling Chen, Stanko Dimitrov, Rahul Sami, Daniel M. Reeves, David M. Pennock, Robin D. Hanson, Lance Fortnow, and Rica Gonen. Gaming prediction markets: Equilibrium strategies with a market maker. Algorithmica, to appear.
• 3: Norman Dalkey and Olaf Helmer. An experimental application of the Delphi method to the use of experts. Management Science, 9, 1963.
• 4: Paul H. Garthwaite, Joseph B. Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470):680--700, 2005.
• 5: Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman&Hall, 2nd edition, 2004.
• 6: Allan Gibbard. Manipulation of voting schemes: A general result. Econometrica, 41:587--601, 1973.
• 7: I.J. Good. Rational decisions. JRSS-B, 1952.
• 8: Jerry R. Green and Jean-Jacques Laffont. Incentives in Public Decision-Making. North-Holland Publishing Company, 1979.
• 9: David J. Johnstone, The Parimutuel Kelly Probability Scoring Rule, Decision Analysis, v.4 n.2, p.66-75, June 2007 [doi>10.1287/deca.1070.0091]
• 10: Radu Jurca , Boi Faltings, Incentives for expressing opinions in online polls, Proceedings of the 9th ACM conference on Electronic commerce, July 08-12, 2008, Chicago, Il, USA [doi>10.1145/1386790.1386812]
• 11: D. Marc Kilgour , Yigal Gerchak, Elicitation of Probabilities Using Competitive Scoring Rules, Decision Analysis, v.1 n.2, p.108-113, June 2004 [doi>10.1287/deca.1030.0003]
• 12: Nicolas S. Lambert , John Langford , Jennifer Wortman , Yiling Chen , Daniel Reeves , Yoav Shoham , David M. Penno k, Self-financed wagering mechanisms for forecasting, Proceedings of the 9th ACM conference on Electronic commerce, July 08-12, 2008, Chicago, Il, USA [doi>10.1145/1386790.1386820]
• 13: Nicolas Lambert , Yoav Shoham, Truthful Surveys, Proceedings of the 4th International Workshop on Internet and Network Economics, December 17-20, 2008, Shanghai, China [doi>10.1007/978-3-540-92185-1_23]
• 14: Andrew Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomics Theory. Oxford University Press, New York, NY, 1995.
• 15: James E. Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science, 22(10):1087--1096, 1976.
• 16: Paul Milgrom and Nancy L. Stokey. Information, trade and common knowledge. Journal of Economic Theory, 26(1):17--27, 1982.
• 17: Nolan Miller , Paul Resnick , Richard Zeckhauser, Eliciting Informative Feedback: The Peer-Prediction Method, Management Science, v.51 n.9, p.1359-1373, September 2005 [doi>10.1287/mnsc.1050.0379]
• 18: A. Murphy and R. Winkler. Probability forecasting in meteorology. Journal of the American Statistical Association, 79(387):489--500, 1984.
• 19: R. Nelson and D. Bessler. Subjective probabilities and scoring rules: Experimental evidence. Journal of Agricultural Economics, 71(2):363--369, 1989.
• 20: F. O'Carroll. Subjective probabilities and short-term economic forecasts: An empirical investigation. Applied Statistics, 26(3):269--278, 1977.
• 21: Drazen Prelec. A Bayesian truth serum for subjective data. Science, 306:462--466, 2004.
• 22: Leonard J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783--801, 1971.
• 23: D. Spiegelhalter. Probabilistic prediction in patient management and clinical trials. Stat. Med., 5:421--433, 1986.
• 24: James Surowiecki, The Wisdom of Crowds, Anchor, 2005
• 25: Robert L. Winkler. The assessment of prior distributions in Bayesian analysis. Journal of the American Statistical Association, 62(319):776--800, 1967.
• 26: Robert L. Winkler. Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66(336):675--685, 1971.
• 27: Robert L. Winkler, J. Muñoz, J.L. Cervera, J.M. Bernardo, G. Blattenberger, J.B. Kadane, D.V. Lindley, A.H. Murphy, R.M. Oliver, and D. Ríos-Insua. Scoring rules and the evaluation of probabilities. TEST/Springer, 5(1):1--60, 1996.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Economics

General Terms:
Algorithms, Design, Economics

Keywords:
forecasting, mechanism design, polls, prediction markets