Use of the Cloze procedure in testing a model of complexity

ABSTRACT

This paper describes an experiment designed and conducted by undergraduate students as part of a senior research project in Computer Science. The experiment tested whether the program comprehension of a set of Fortran 77 programs can be predicted by the Oviedo/Van Verth model of program complexity. In the study, student programmers were asked to demonstrate their ability to understand programs through use of the cloze procedure. Three different kinds of Fortran 77 programs were used at three different levels of complexity determined by the model. It was hypothesized that program comprehension would decrease, i.e. the number of incorrect answers would increase, as the complexity of the programs increased. Due to a variety of reasons, the results were inconclusive. The purpose of this paper is to discuss the experiment, the method selected, and implications for future experiments.

REFERENCES

• BRO80: Ruven E. Brooks, Studying programmer behavior experimentally: the problems of proper methodology, Communications of the ACM, v.23 n.4, p.207-213, April 1980 [doi>10.1145/358841.358847]
• CUR79a: Bill Curtis , Sylvia B. Sheppard , Phil Milliman, Third time charm: Stronger prediction of programmer performance by software complexity metrics, Proceedings of the 4th international conference on Software engineering, p.356-360, September 17-19, 1979, Munich, Germany
• CUR79b: Curti~ B. Sheppard, S~ Milliman, P. Borst, M~ & Love, T~ ~'Measuring the Psychological Complexity of Soft-ware Maintenance Tasks with the Ha~tead and McC~be Metrics", IEEE TSE, Vol. SE-5, No~ 2, March, 1979, pp. 96-104.
• ENT84: Eileen B. Entin, Using the cloze procedure to assess program reading comprehension, ACM SIGCSE Bulletin, v.16 n.1, p.44-50, February 1984
• ENT86: Eileen B. Entin, Using the cloze procedure with computer programs: a deeper look, ACM SIGCSE Bulletin, v.18 n.1, p.153-162, February 1986
• HAL86: William E. Hall, III , Stuart H. Zweben, The Cloze procedure and software comprehensibility measurement, IEEE Transactions on Software Engineering, v.12 n.5, p.608-623, May 1986
• KIR68: Kirk, RiE~ Experimental Design ~ Procedures for the Behavioral Sciences. Belmont : Wadsworth Inc~ 1968.
• OVI80: Oviedo, F~ ~'Control Flow, Data Flow and Program Complexity", COMPSAC, December, 1980, pp. 146- 152.
• SHE79: Sheppard, S, Curtis, B~ Milliman, P, &, Love, T~ "Modern C_.xxting Practices and Programmer Performance", Computer, VoL 12, N~ 12, December, 1979, pp. 41-49.
• TAY53: Taylor, W. L, "Cloze proc~ure: A new tool for measuring readability", Jouraaiism Quarterly, Vol. 30, No. 4, Fall 1953, pl~ 415-63.
• THO86: Mark Thomas , Stuart Zweben, The effects of program-dependent and program-independent deletions on software cloze tests, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.138-152, June 1986, Washington, D.C., United States
• VAN87: Van Verth, Patricia, ~'A Program Complexity Model That Includes Procedures", COMPSAC, October 1987, pp. 252-258.
• WEI74: Larry Weissman, Psychological complexity of computer programs: an experimental methodology, ACM SIGPLAN Notices, v.9 n.6, p.25-36, June 1974 [doi>10.1145/953233.953237]

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Curriculum

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures

General Terms:
Design, Experimentation, Human Factors, Management, Measurement, Performance