|
 ABSTRACT
Previous studies report that software developers are over-confident in the accuracy of their effort estimates. Aim: This study investigates the role of outcome feedback, that is, feedback about the discrepancy between the estimated and the actual effort, in improving the uncertainty assessments. Method: We conducted two in-depth empirical studies on uncertainty assessment learning. Study 1 included five student developers and Study 2, 10 software professionals. In each study the developers repeatedly assessed the uncertainty of their effort estimates of a programming task, solved the task, and received estimation accuracy outcome feedback. Results: We found that most, but not all, developers were initially over-confident in the accuracy of their effort estimates and remained over-confident in spite of repeated and timely outcome feedback. One important, but not sufficient, condition for improvement based on outcome feedback seems to be the use of explicitly formulated, instead of purely intuition-based, uncertainty assessment strategies.
REFERENCES
Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.
| |
1
|
Bengtsson, C., Persson, M., and Willenhag, P. 2005. Gender and overconfidence. Economics Lett. 86, 2 (Feb.), 199--203.
|
| |
2
|
|
| |
3
|
Bryan, G. E. 1994. Not All Programmers Are Created Equal. In Proceedings of the IEEE Aerospace Applications Conference, 55--62.
|
| |
4
|
Buehler, R. and Griffin, D. 2003. Planning, personality, and prediction: The role of future focus in optimistic time predictions. Organiza. Behav. Hum. Decision Proc. 92, 80--90.
|
| |
5
|
Buehler, R., Griffin, D., and Ross, M. 1994. Exploring the “Planning fallacy”: Why people underestimate their task completion times. J. Personality Soc. Psych. 67, 3, 366--381.
|
| |
6
|
Buehler, R., Griffin, D., and MacDonald, H. 1997. The role of motivated reasoning in optimistic time predictions. Personality Soc. Psych. Bull. 23, 3, 238--247.
|
| |
7
|
|
| |
8
|
Einhorn, H. J. and Hogarth, R. M. 1978. Confidence in judgment: Persistence of the illusion of validity. Psych. Rev. 85, 5, 395--416.
|
| |
9
|
Ericson, K. A. and Krampe, R. T. 1993. The role of deliberate practice in the acquisition of expert performance. Psych. Rev. 100, 3, 363--406.
|
| |
10
|
Gagne, R. M. and Medsker, K. L. 1996. The Conditions of Learning: Training Applications. Harcourt Brace.
|
| |
11
|
Gigerenzer, G. and Hoffrage, U. 1995. How to improve Bayesian reasoning without instruction: Frequency formats. Psych. Rev. 102, 4, 684--704.
|
| |
12
|
|
| |
13
|
Green, K. W. J., Medlin, B., and Whitten, D. 2004. Developing optimism to improve performance: an approach for the manufacturing sector. Industrial Manag. Data Sys. 104, 2, 106--114.
|
| |
14
|
Hammond, K. R. 1996. Human Judgement and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. Oxford University Press, Oxford, UK.
|
| |
15
|
|
| |
16
|
Heine, S. J. and Lehman, D. R. 1995. Cultural variation in unrealistic optimism: Does the West feel more invulnerable than the East? J. Personality Soc. Psych. 68, 4, 595--607.
|
| |
17
|
Hogarth, R. M. 2001. Educating Intuition. University of Chicago Press, Chicago.
|
| |
18
|
Hughes, R. T. 1996. Expert judgement as an estimating method. Inform. Softw. Tech. 38, 2, 67--75.
|
| |
19
|
Humphrey, W. S. 1996. The PSP and personal project estimating. Amer. Program. 9, 6, 2--15.
|
| |
20
|
Johnson, D. D. P. 2004. Overconfidence and War. Harvard University Press, Cambridge, MA.
|
| |
21
|
Jørgensen, M. 2004a. Top-down and bottom-up expert estimation of software development effort. Inform. Softw. Tech. 46, 1, 3--16.
|
| |
22
|
Jørgensen, M. 2004b. A review of studies on expert estimation of software development effort. J. Syst. Softw. 70, 1--2, 37--60.
|
| |
23
|
|
| |
24
|
|
| |
25
|
Jørgensen, M. and Moløkken-Østvold, K. 2004a. Eliminating over-confidence in software development effort estimates. In Conference on Product Focused Software Process Improvement (PROFES) Springer-Verlag 174--184.
|
| |
26
|
|
| |
27
|
|
| |
28
|
Jørgensen, M. and Sjøberg, D. I. K. 2003a. An effort prediction interval approach based on the empirical distribution of previous estimation accuracy. Inform. Softw. Tech. 45, 3, 123--136.
|
| |
29
|
Jørgensen, M. and Sjøberg. D. 2003b. Generalization and theory-building in software engineering research. In Proceedings of Empirical Assessment in Software Engineering (EASE'04) 29--36.
|
| |
30
|
Jørgensen, M., Faugli, B., and Gruschke, T. M. Characteristics of Software Engineers with Optimistic Predictions. J. Syst. Softw. To appear.
|
| |
31
|
|
| |
32
|
Jørgensen, M., Teigen, K. H., and Moløkken, K. 2004. Better sure than safe? Over-confidence in judgement based software development effort prediction intervals. J. Syst. Softw. 70, 1-2, 79--93.
|
| |
33
|
|
| |
34
|
Klayman, J., Soll, J. B., González-Vallejo, C., and Barlas, S. 1999. Overconfidence: It depends on how, what and whom you ask. Organizat. Behav. Hum. Decision Proc. 79, 3, 216--247.
|
| |
35
|
Kruglanski, A. W. 1996. Motivated social cognition: Principles of the interface. In Social Psychology: Handbook of Basic Principles, E.T. Higgins and A. W. Kurglanski, Eds. Guilford Press, New York, 493--520.
|
| |
36
|
Li, H., Shen, Q. P., and Love, P. E. D. 2005. Cost modelling of office buildings in Hong Kong: An exploratory study. Facilities 23, 9/10, 438--452.
|
| |
37
|
Li, S. and Fang, Y. 2004. Respondents in Asian cultures (e.g., Chinese) are more risk-seeking and more overconfident than respondents in other cultures (e.g., in United States) but the reciprocal predictions are in total opposition: how and why? J. Cognition Culture 4, 2, 263--292.
|
| |
38
|
Lichtenstein, S. and Fischhoff, B. 1977. Do those who know more also know more about how much they know? Organizat. Behav. Hum. Decision Proc. 20, 2, 159--183.
|
| |
39
|
Manove, M. 1995. Entrepeneurs, optimism and the competitive edge. UFAE and IAE Working Papers 296.95,
|
| |
40
|
|
| |
41
|
|
| |
42
|
|
| |
43
|
Oskamp, S. 1965. Overconfidence in case-study judgments. J. Consult. Psych. 29, 3, 261--265.
|
 |
44
|
|
| |
45
|
|
| |
46
|
Puncochar, J. M. and Fox, P. W. 2004. Confidence in individual and group decision making: When “two heads” are worse than one. J. Educat. Psych. 96, 3, 582--591.
|
| |
47
|
Rozenblit, L. and Keil, F.C. 2002. The misunderstood limits of folk science: an illusion of explanatory depth. Cognitive Sci. 26, 5, 521--562.
|
| |
48
|
Shadish, W. R., Cook, T. D., and Campbell, D.T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.
|
| |
49
|
|
| |
50
|
Shepperd, J. A., Carroll, P., Grace, J., and Terry, M. 2002. Exploring the causes of comparative optimism. Psychologica Belgica 42, 1/2, 65--98.
|
| |
51
|
Shepperd, J. A., Fernandez, J. K. and Quellette, J. A. 1996. Abandoning unrealistic optimism: Performance estimates and the temporal proximity of self-relevant feedback. J. Personality Social Psych. 70, 4, 844--855.
|
| |
52
|
Steen, v. d. E. 2004. Rational overoptimism (and other biases). Amer. Econ. Rev. 94, 4, 1141--1151.
|
| |
53
|
Webster Jr, M. 1994. Experimental methods. In Group Processes, M. Foschi and E. J. Lawler, Eds., Nelson-Hall Publishers, 43--69.
|
| |
54
|
Whitecotton, S. M., Sanders, D. E., and Norris, K. B. 1998. Improving predictive accuracy with a combination of human intuition and mechanical decision aids. Organizat. Behav. Hum. Decision Proces. 76, 3, 325--348.
|
| |
55
|
Wilson, L. F. and Brekke, N. 1994. Mental contamination and mental correction: unwanted influence on judgments and evaluation. Psych. Bull. 116, 1, 117--142.
|
| |
56
|
Zendler, A. E., Horn, E., Schwartzel, H., and Plodererder, E. 2001. Demonstrating the usage of single-case designs in experimental software engineering. Inform. Softw. Tech. 43, 12, 681--691.
|
|
Adobe Acrobat
QuickTime
Windows Media Player
Real Player
Pdf
D.2