Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

This paper describes the use of random code generation and mutation as a method for synthesising multiple choice questions which can be used in automated assessment. Whilst using multiple choice questions has proved to be a feasible method of testing if students have suitable knowledge or comprehension of a programming concept, creating suitable multiple choice questions that accurately test the students' knowledge is time intensive.This paper proposes two methods of generating code which can then be used to closely examine the comprehension ability of students. The first method takes as input a suite of template programs, and performs slight mutations on each program and ask students to comprehend the new program. The second method performs traversals on a syntax tree of possible programs, yielding slightly erratic but compilable code, again with behaviour that students can be questioned about. As well as generating code these methods also yield alternative distracting answers to challenge the students. Finally, this paper discusses the gradual introduction of these automatically generated questions as an assessment method and discusses the relative merits of each technique.

REFERENCES

• 1: R. F. Biehler and J. Snowman. Psychology Applied to Teaching. Houghton Mifflin Company, 8th edition, 1997.
• 2: B. S. Bloom and D. R. Krathowl. Taxonomy of educational objectives. McKay & Co, 1956.
• 3: Charlie Daly , John Waldron, Assessing the assessment of programming ability, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 4: B. duBoulay. Some difficulties of learning to program. In E. Soloway and J. C. Spohrer, editors, Studying the Novice Programmer, chapter 15, pages 283--301. Lawrence Erlbaum Associates, 1989.
• 5: Ann E. Fleury, Programming in Java: student-constructed rules, Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.197-201, March 07-12, 2000, Austin, Texas, United States
• 6: Maria Hristova , Ananya Misra , Megan Rutter , Rebecca Mercuri, Identifying and correcting Java programming errors for introductory computer science students, Proceedings of the 34th SIGCSE technical symposium on Computer science education, February 19-23, 2003, Reno, Navada, USA
• 7: Raymond Lister, On blooming first year programming, and its blooming assessment, Proceedings of the Australasian conference on Computing education, p.158-162, December 2000, Melbourne, Australia [doi>10.1145/359369.359393]
• 8: Raymond Lister, Objectives and objective assessment in CS1, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.292-296, February 2001, Charlotte, North Carolina, United States
• 9: Raymond Lister , John Leaney, Introductory programming, criterion-referencing, and bloom, Proceedings of the 34th SIGCSE technical symposium on Computer science education, February 19-23, 2003, Reno, Navada, USA
• 10: Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer , Lynda Thomas , Ian Utting , Tadeusz Wilusz, A multi-national, multi-institutional study of assessment of programming skills of first-year CS students, ACM SIGCSE Bulletin, v.33 n.4, December 2001 [doi>10.1145/572139.572181]
• 11: Nelishia Pillay, Using genetic programming for the induction of novice procedural programming solution algorithms, Proceedings of the 2002 ACM symposium on Applied computing, March 11-14, 2002, Madrid, Spain [doi>10.1145/508791.508903]
• 12: R. Putnam, D. Sleeman, J. Baxter and, and L. Kuspa. A summary of misconceptions of high school BASIC programmers. Journal of Educational Computing Research, 2:459--472, 1986.
• 13: Anthony Rhodes , Karyn Bower , Peter Bancroft, Managing large class assessment, Proceedings of the sixth conference on Australasian computing education, p.285-289, January 01, 2004, Dunedin, New Zealand
• 14: Conor Ryan , J. J. Collins , Michael O'Neill, Grammatical Evolution: Evolving Programs for an Arbitrary Language, Proceedings of the First European Workshop on Genetic Programming, p.83-96, April 14-15, 1998
• 15: James C. Spohrer , Elliot Soloway, Novice mistakes: are the folk wisdoms correct?, Communications of the ACM, v.29 n.7, p.624-632, July 1986 [doi>10.1145/6138.6145]
• 16: R. C. Sprinthall, N. A. Sprinthall, and S. N. Oja. Educational Psycholgy. McGraw-Hill Education, 7th edition, 1998.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

Additional Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Self-assessment

General Terms:
Design, Human Factors

Keywords:
assessment, first year programming, program comprehension