A multi-national study of reading and tracing skills in novice programmers

ABSTRACT

A study by a ITiCSE 2001 working group ("the McCracken Group") established that many students do not know how to program at the conclusion of their introductory courses. A popular explanation for this incapacity is that the students lack the ability to problem-solve. That is, they lack the ability to take a problem description, decompose it into sub-problems and implement them, then assemble the pieces into a complete solution. An alternative explanation is that many students have a fragile grasp of both basic programming principles and the ability to systematically carry out routine programming tasks, such as tracing (or "desk checking") through code. This ITiCSE 2004 working group studied the alternative explanation, by testing students from seven countries, in two ways. First, students were tested on their ability to predict the outcome of executing a short piece of code. Second, students were tested on their ability, when given the desired function of short piece of near-complete code, to select the correct completion of the code from a small set of possibilities. Many students were weak at these tasks, especially the latter task, suggesting that such students have a fragile grasp of skills that are a prerequisite for problem-solving.

REFERENCES

• 1: Brooks, R. (1983) Towards a theory of the comprehension of computer programs. International Journal of man-Machine Studies, 18, pp. 543--554.
• 2: Michael J. Clancy , Marcia C. Linn, Patterns and pedagogy, The proceedings of the thirtieth SIGCSE technical symposium on Computer science education, p.37-42, March 24-28, 1999, New Orleans, Louisiana, United States
• 3: Davies, S. (1996) Display-based problem solving strategies in computer programming. In Gray, W, and Boehm-Davis, D. (Eds) Empirical Studies of Programmers: 6th Workshop. Ablex Publishing Corporation, NJ. pp. 59--76.
• 4: Detienne, F. (1990) Expert Programming Knowledge: A Schema-based Approach. In Hoc, J, Green, T, Samurcay, and Gilmore, D. (Eds) Psychology of Programming. Academic Press, London. pp 206--222.
• 5: Françoise Détienne, An empirically-derived control structure for the process of program understanding, International Journal of Man-Machine Studies, v.33 n.3, p.323-342, Sept. 1990 [doi>10.1016/S0020-7373(05)80122-1]
• 6: Deimel, L. E. & Naveda, J. F. (1990) Reading Computer Programs: Instructor's Guide and Exercises Software Engineering Institute, Carnegie-Mellon University. http://www.deimel.org/comp_sci/reading_computer_programs.htm (August 2004) Updated bibliography at http://www2.umassd.edu/SWPI/Process Bibliography/bib-codereading2.html (August 2004).
• 7: du Boulay, B. (1989) Some Difficulties of Learning to Program. In Soloway, E. and Spohrer, J., Eds. pp. 283--299.
• 8: Ebel, R. and Frisbie, D. (1986) Essentials of Educational Measurement. Prentice Hall, Englewood Cliffs, NJ.
• 9: Haladyna, T. (1999) Developing and Validating Multiple-Choice Questions (2nd Edition), Lawrence Erlbaum Associates, Mahwah, NJ.
• 10: Marja Kuittinen , Jorma Sajaniemi, Teaching roles of variables in elementary programming courses, Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education, June 28-30, 2004, Leeds, United Kingdom
• 11: Linn, R. and Gronlund, N. (1995) Measurement and Assessment in Teaching, Prentice Hall, Upper Saddle River, NJ.
• 12: Lister, R. (2004) Availability of working group data. http://www-staff.it.uts.edu.au/~raymond/leeds2004.
• 13: Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer , Lynda Thomas , Ian Utting , Tadeusz Wilusz, A multi-national, multi-institutional study of assessment of programming skills of first-year CS students, ACM SIGCSE Bulletin, v.33 n.4, December 2001 [doi>10.1145/572139.572181]
• 14: Nancy Pennington, Comprehension strategies in programming, Empirical studies of programmers: second workshop, Ablex Publishing Corp., Norwood, NJ, 1987
• 15: D. N. Perkins , Fay Martin, Fragile knowledge and neglected strategies in novice programmers, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.213-229, June 1986, Washington, D.C., United States
• 16: Perkins, D, Hancock, C, Hobbs, R, Martin, F, and Simmons, R. (1989). Conditions of Learning in Novice Programmers. In Soloway, E. and Spohrer, J., Eds. pp 261--279.
• 17: Robert S. Rist, Plans in programming: definition, demonstration, and development, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.28-47, June 1986, Washington, D.C., United States
• 18: Rist, R. (2004) Learning to Program: Schema Creation, Application, and Evaluation. In Fincher, S and Petre, M., Eds (2004) Computer Science Education Research. Swets & Zeitlinger.
• 19: Soloway, E. and Ehrlich, K (1984) Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering, SE-10(5):595--609.
• 20: E. Soloway, Learning to program = learning to construct mechanisms and explanations, Communications of the ACM, v.29 n.9, p.850-858, Sept. 1986 [doi>10.1145/6592.6594]
• 21: Elliot Soloway , Ben Shneiderman , Sitharama Iyengar, Empirical Studies of Programmers: First Workshop, Greenwood Publishing Group Inc., Westport, CT, 1986
• 22: Soloway, E, Adelson, B, and Ehrlich, K. (1988) Knowledge and Processes in the Comprehension of Computer Programs. In Glaser, M, Chi, R, Farr, M, Glaser, R (Eds) The Nature of Expertise. Lawrence Erlbaum Associates, Hillsdale, NJ, USA. pp 129--152.
• 23: E. Soloway , James C. Spohrer, Studying the Novice Programmer, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1988
• 24: James G. Spohrer , Elliot Soloway, Analyzing the high frequency bugs in novice programs, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.230-251, June 1986, Washington, D.C., United States
• 25: Spohrer, J. and Soloway, E. (1989) Novice Mistakes: Are the Folk Wisdoms Correct? In Soloway, E. and Spohrer, J., Eds. pp 401--416.
• 26: Lynda Thomas , Mark Ratcliffe , Benjy Thomasson, Scaffolding with object diagrams in first year programming classes: some unexpected results, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 27: Susan Wiedenbeck, Novice/expert differences in programming skills, International Journal of Man-Machine Studies, v.23 n.4, p.383-390, Oct. 1985 [doi>10.1016/S0020-7373(85)80041-9]
• 28: Susan Weidenbeck, Processes in computer program comprehension, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.48-57, June 1986, Washington, D.C., United States