Intention-based scoring: an approach to measuring success at solving the composition problem

ABSTRACT

Traditional methods of evaluating student programs are not always appropriate for assessment of different instructional interventions. They tend to focus on the final product rather than on the process that led to it. This paper presents intention-based scoring (IBS), an approach to measuring programming ability that looks at intermediate programs produced over the course of an implementation rather than just the one at the end. The intent is to assess a student's ability to produce algorithmically correct code on the first attempt at achieving each program goal. In other words, the goal is to answer question "How close was the student to being initially correct?" but not to speak to a student's debugging skills or ability to ultimately produce a working program. To produce an IBS, it is necessary to inspect a student's online protocol, which is simply the collection of all programs submitted to a compiler. IBS involves a three-phase process of (1) identification of the subset of all programs in a protocol that represent the initial attempts at achieving programming goals, (2) bug identification, and (3) rubric-based scoring. We conclude with an example application of IBS in the evaluation of a tutoring system for beginning programmers and also show how an IBS can be broken down by the underlying bug categories to reveal more subtle differences.

REFERENCES

• 1: Charlie Daly , John Waldron, Assessing the assessment of programming ability, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 2: Wayne D. Gray , John R. Anderson, Change-episodes in coding: when and how do programmers change their code?, Empirical studies of programmers: second workshop, Ablex Publishing Corp., Norwood, NJ, 1987
• 3: M. Guzdial, L. Hohmann, M. Konneman, C. Walton, and E. Soloway. Supporting programming and learning-to-program with an integrated cad and scaffolding workbench. Interactive Learning Environments, 6(1&2):143--179, 1998.
• 4: M. C. Jadud. A first look at novice compilation behavior using bluej. In 16th Annual Workshop of the Psychology of Programming Interest Group (PPIG 2004), Institute of Technology, Carlow, Ireland, April 2004.
• 5: W. L. Johnson, Understanding and debugging novice programs, Artificial Intelligence, v.42 n.1, p.51-97, Feb. 1990 [doi>10.1016/0004-3702(90)90094-G]
• 6: H. C. Lane and K. VanLehn. A dialogue-based tutoring system for beginning programming. In Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS), pages 449--454, Miami Beach, FL, 2004. AAAI Press.
• 7: H. C. Lane and K. VanLehn. Teaching program planning skills to novices with natural language tutoring. In S. Fitzgerald and M. Guzdial, editors, Computer Science Education. Swets and Zeitlinger, September 2005. Special issue on doctoral research in CS Education.
• 8: Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer , Lynda Thomas , Ian Utting , Tadeusz Wilusz, A multi-national, multi-institutional study of assessment of programming skills of first-year CS students, ACM SIGCSE Bulletin, v.33 n.4, December 2001 [doi>10.1145/572139.572181]
• 9: R. S. Rist. Program Structure and Design. Cognitive Science, 19:507--562, 1995.
• 10: E. Soloway and K. Ehrlich. Empirical studies of programming knowledge. IEEE Transactions on Software and Engineering, SE-10(5):595--609, September 1984.
• 11: E. Soloway , James C. Spohrer, Studying the Novice Programmer, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1988
• 12: J. C. Spohrer and E. Soloway. Putting it all together is hard for novice programmers. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Tucson, Arizona, November 12-15 1985.
• 13: J. C. Spohrer, E. Soloway, and E. Pope. A goal/plan analysis of buggy pascal programs. In Soloway and Spohrer {11}, pages 355--399.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

General Terms:
Measurement

Keywords:
intelligent tutoring systems, intention-based scoring, novice programming, online protocols, structured programming