Eliciting properties of probability distributions

ABSTRACT

We investigate the problem of truthfully eliciting an expert's assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or "score") functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

REFERENCES

• 1: G. Brier. Verification of Forecasts Expressed In Terms of Probability. Monthly Weather Review, 78(1):1--3, 1950.
• 2: A. Buja, W. Stuetzle, and Y. Shen. Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications. Working paper, 2005.
• 3: J. Cervera and J. Munoz. Proper scoring rules for fractiles. Bayesian Statistics, 5:513--519, 1996.
• 4: Joan Feigenbaum , Lance Fortnow , David M. Pennock , Rahul Sami, Computation in a distributed information market, Theoretical Computer Science, v.343 n.1-2, p.114-132, 10 October 2005 [doi>10.1016/j.tcs.2005.05.010]
• 5: R. Forsythe, F. Nelson, G. Neumann, and J. Wright. Anatomy of an Experimental Political Stock Market. The American Economic Review, 82(5):1142--1161, 1992.
• 6: D. Friedman. Effective Scoring Rules for Probabilistic Forecasts. Management Science, 29(4):447--454, 1983.
• 7: T. Gneiting and A. Raftery. Strictly proper scoring rules, prediction and estimation. Journal of the American Statistical Association, 102(477):359--378, 2007.
• 8: I. Good. Rational Decisions. Springer-Verlag, 1997.
• 9: Robin Hanson, Combinatorial Information Market Design, Information Systems Frontiers, v.5 n.1, p.107-119, January 2003 [doi>10.1023/A:1022058209073]
• 10: A. Hendrickson and R. Buehler. Proper Scores for Probability Forecasters. The Annals of Mathematical Statistics, 42(6):1916--1921, 1971.
• 11: A. Murphy and R. Winkler. Probability Forecasting in Meterology. Journal of the American Statistical Association, 79(387):489--500, 1984.
• 12: R. Nelson and D. Bessler. Subjective Probabilities and Scoring Rules: Experimental Evidence. American Journal of Agricultural Economics, 71(2):363--369, 1989.
• 13: F. O'Carroll. Subjective Probabilities and Short-Term Economic Forecasts: An Empirical Investigation. Applied Statistics, 26(3):269--278, 1977.
• 14: L. Savage. Elicitation of Personal Probabilities and Expectations. Journal of the American Statistical Association, 66(336):783--801, 1971.
• 15: M. Schervish. A General Method for Comparing Probability Assessors. The Annals of Statistics, 17(4):1856--1879, 1989.
• 16: E. Shuford, A. Albert, and H. Edward Massengill. Admissible probability measurement procedures. Psychometrika, 31(2):125--145, 1966.
• 17: D. Spiegelhalter. Probabilistic prediction in patient management and clinical trials. Stat Med, 5(5):421--33, 1986.
• 18: C. Stael von Holstein. A Family of Strictly Proper Scoring Rules Which Are Sensitive to Distance. Journal of Applied Meteorology, 9(3):360--364, 1970.
• 19: R. Winkler. Scoring rules and the evaluation of probabilities. TEST, 5(1):1--60, 1996.
• 20: J. Wolfers and E. Zitzewitz. Prediction Markets. The Journal of Economic Perspectives, 18(2):107--126, 2004.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Economics

General Terms:
Economics, Theory

Keywords:
elicitation, forecasting, prediction market, scoring rule