Studying programmer behavior experimentally: the problems of proper methodology

ABSTRACT

The application of behavioral or psychological techniques to the evaluation of programming languages and techniques is an approach which has found increased applicability over the past decade. In order to use this approach successfully, investigators must pay close attention to methodological issues, both in order to insure the generalizability of their findings and to defend the quality of their work to researchers in other fields. Three major areas of methodological concern, the selection of subjects, materials, and measures, are reviewed. The first two of these areas continue to present major difficulties for this type of research.

REFERENCES

• 1: Boies, S.F., and Gould, J.D. Syntactic errors in computer programming. Human Factors 16 (1974), 253-257.
• 2: Ruven Brooks, Using a behavioral theory of program comprehension in software engineering, Proceedings of the 3rd international conference on Software engineering, p.196-201, May 10-12, 1978, Atlanta, Georgia, United States
• 3: Chase, W.G., and Simon, H.A. Perception in chess. Cognitive Psychol. 4 (1974), 55-81.
• 4: Bill Curtis , Sylvia B. Sheppard , Phil Milliman, Third time charm: Stronger prediction of programmer performance by software complexity metrics, Proceedings of the 4th international conference on Software engineering, p.356-360, September 17-19, 1979, Munich, Germany
• 5: Garmon, J.D. An experiment for the evaluation of language features. Internat. J. of Man-Machine Studies 8 (1976), 61-73.
• 6: J. D. Gannon, An experimental evaluation of data type conventions, Communications of the ACM, v.20 n.8, p.584-595, Aug. 1977 [doi>10.1145/359763.359800]
• 7: Gould, J.D., and Drongowski, P. An exploratory study of computer program debugging. Human Factors 16 (1974), 258-276.
• 8: Gould, J.D. Some psychological evidence on how people debug computer programs, lnternat. J. of Man-Machine Studies 7 (1975), 151-182.
• 9: Green, T.R.G. Conditional program statements and their comprehensibility to professional programmers. J. of Occupational Psychol. 50 (1977), 93-109.
• 10: Maurice H. Halstead, Elements of Software Science (Operating and programming systems series), Elsevier Science Inc., New York, NY, 1977
• 11: Knuth, D.E. An empirical study of FORTRAN programs. Software Practice and Experience 1, 2 (1971) 105-133.
• 12: McCabe, T.J. A complexity measure. 1EEE Trans. on Software Eng. SE-2 (1976), 308-320.
• 13: Allen Newell, Human Problem Solving, Prentice-Hall, Inc., Upper Saddle River, NJ, 1972
• 14: Rubey, R.J. A comparative evaluation of PL/1. Datamation 20 (Dec. 1968).
• 15: Sackman, H. Man-Computer Problem Solving: Experimental Evaluation of Time-Sharing and Batch Processing. Auerbach, New York, 1970.
• 16: Sheppard, S.B., Borst, M.A., Curtis, B., and Love, L.T. Predicting programmers' ability to modify software. TR-388100-3, General Electric Co., Arlington, Va., 1978.
• 17: Sheppard, S.B., Borst, M.A., Millman, P., and Love, T. Modern coding practices and programmer performance. Computer 12 (1979).
• 18: Ben Shneiderman , Richard Mayer , Don McKay , Peter Heller, Experimental investigations of the utility of detailed flowcharts in programming, Communications of the ACM, v.20 n.6, p.373-381, June 1977 [doi>10.1145/359605.359610]
• 19: Shneiderman, B. Measuring computer program quality and comprehension, lnternat. J. of Man-Machine Studies 9 (1977), 465- 478.
• 20: Sime, M.E., Arblaster, A.T., and Green, T.R.G. Reducing programming errors in nested conditionals by prescribing a writing procedure. Internat. J. of Man-Machine Studies 9 (1977), 119-126.
• 21: Sime, M.E., Green, T.R.G., and Guest, D.J. Psychological evaluation of two conditional constructions used in computer languages, lnternat. J. of Man-Machine Studies 5 (1973), 105-113.
• 22: Sime, M.E., Green, T.R.G., and Guest, D.J. Scope marking in computer conditionals--a psychological evaluation. Internat. J. of Man-Machine Studies 9 (1977), 107-118.
• 23: Weinberg, G., and Shulman, E. Goals and performance in computer programming. Human Factors 16, 1 (1974).
• 24: Laurence Mark Weissman, A methodology for studying the psychological complexity of computer programs., 1974
• 25: Winer, B. Statistical Principles in Experimental Design. McGraw- Hill, New York, 1971.
• 26: Youngs, E.A. Human errors in programming, lnternat. J. of Man-Machine Studies 6 (1974), 361-376.

INDEX TERMS

Keywords:
psychology of programming, software psychology