Individualized exercises for self-assessment of programming knowledge: An evaluation of QuizPACK

ABSTRACT

Individualized exercises are a promising feature in promoting modern e-learning. The focus of this article is on the QuizPACK system, which is able to generate parameterized exercises for the C language and automatically evaluate the correctness of student answers. We introduce QuizPACK and present the results of its comprehensive classroom evaluation during four consecutive semesters. Our studies demonstrate that when QuizPACK is used for out-of-class self-assessment, it is an exceptional learning tool. The students' work with QuizPACK significantly improved their knowledge of semantics and positively affected higher-level knowledge and skills. The students themselves praised the system highly as a learning tool. We also demonstrated that the use of the system in self-assessment mode can be significantly increased by basing later classroom paper-and-pencil quizzes on QuizPACK questions, motivating students to practice them more.

REFERENCES

• 1: Arnow, D. and Barshay, O. 1999. WebToTeach: A web-based automated program checker. In Proceedings of the Frontiers in Education Conference (FIE99, San Juan, Puerto Rico).
• 2: Benford, S. D., Burke, E. K., Foxley, E., and Higgins, C. A. 1994. Ceilidh: A courseware system for the assessment and administration of computer programming courses in higher education. In Proceedings of the Interdisciplinary Workshop on Complex Learning in Computer Environments (CLCE94, Joensuu, Finland).
• 3: Bloom, B. S. 1956. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. David McKay Co. Inc., New York.
• 4: Bloom, B. S. 1984. The 2 sigma problem: The search for methods of group instruction as effective one-to-one tutoring. Edu. Res. 13, 3.
• 5: Stina Bridgeman , Michael T. Goodrich , Stephen G. Kobourov , Roberto Tamassia, PILOT: an interactive tool for learning and grading, Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.139-143, March 07-12, 2000, Austin, Texas, United States
• 6: Brusilovsky, P. 1992. Intelligent tutor, environment and manual for introductory programming. Edu. Training Tech. Int. 29, 1, 26--34.
• 7: Brusilovsky, P. 1994. ILEARN: An intelligent system for teaching and learning about UNIX. In Proceedings of the SUUG International Open Systems Conference (Moscow), 35--41.
• 8: Peter Brusilovsky, KnowledgeTree: a distributed architecture for adaptive e-learning, Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, May 19-21, 2004, New York, NY, USA [doi>10.1145/1013367.1013386]
• 9: Peter Brusilovsky , Eduardo Calabrese , Jozef Hvorecky , Anatoly Kouchnirenko , Philip Miller, Mini-languages: a way to learn programming principles, Education and Information Technologies, v.2 n.1, p.65-83, 1997- 1998 [doi>10.1023/A:1018636507883]
• 10: Brusilovsky, P. and Miller, P. 1999. Web-based testing for distance education. In Proceedings of the WebNet'99 World Conference of the WWW and Internet (Honolulu, HI), 149--154.
• 11: Brusilovsky, P. and Miller, P. 2001. Course delivery systems for the virtual university. In: Access to Knowledge: New Information Technologies and the Emergence of the Virtual University. T. Tschang and T. Della Senta, eds. Elsevier Science, Amsterdam, 167--206.
• 12: Peter Brusilovsky , Sergey Sosnovsky, Engaging students to work with self-assessment questions: a study of two approaches, Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education, June 27-29, 2005, Caparica, Portugal
• 13: Charlie Daly, RoboProf and an introductory computer programming course, ACM SIGCSE Bulletin, v.31 n.3, p.155-158, Sept. 1999
• 14: Dancik, G. and Kumar, A. N. 2003. A tutor for counter-controlled loop concepts and its evaluation. In Proceedings of the 2003 Frontiers in Education Conference (FIE 2003, Boulder, CO), Session T3C.
• 15: Dempster, A. P., Laird, N. M., and Rubin, D. 1977. Maximum likelihood estimation from incomplete data via the EM algorithm. J. Royal Stat. Soc. 39, 1, 1--38.
• 16: Eric Fernandes , Amruth N. Kumar, A tutor on scope for the programming languages course, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 17: Garner, S. 2002. Reducing the cognitive load on novice programmers. In Proceedings of the ED-MEDIA'2002---World Conference on Educational Multimedia, Hypermedia and Telecommunications (Denver, CO), 578--583.
• 18: Graham, C. R., Swafford, M. L., and Brown, D. J. 1997. Mallard: A Java enhanced learning environment. In Proceedings of the WebNet'97 World Conference of the WWW, Internet and Intranet (Toronto, Ont.), 634--636.
• 19: Higgins, C., Gray, G., Symeonidis, P., and Tsintsifas, A. 2006. Experiences on automatically assessed algorithmic exercises. ACM J. Edu. Resources Comput. In this issue.
• 20: Colin Higgins , Tarek Hegazy , Pavlos Symeonidis , Athanasios Tsintsifas, The CourseMarker CBA System: Improvements over Ceilidh, Education and Information Technologies, v.8 n.3, p.287-304, September 2003 [doi>10.1023/A:1026364126982]
• 21: David Jackson , Michelle Usher, Grading student programs using ASSYST, ACM SIGCSE Bulletin, v.29 n.1, p.335-339, March 1997
• 22: Mike Joy , Nathan Griffiths , Russell Boyatt, The boss online submission and assessment system, Journal on Educational Resources in Computing (JERIC), v.5 n.3, p.2-es, September 2005 [doi>10.1145/1163405.1163407]
• 23: Kashy, D. A., Albertelli, G., Ashkenazy, G., Kashy E., Ng, H.-K., and Theonnessen, M. 2001. Individualized interactive exercises: A promising role for network technology. In Proceedings of the 31st ASEE/IEEE Frontiers in Education Conference (Reno, NV).
• 24: Kashy, E., Theonnessen, M., Tsai, Y., Davis, N. E., and Wolfe, S. L. 1997. Using networked tools to enhance student success rates in large classes. In Proceedings of the 27th ASEE/IEEE Frontiers in Education Conference (Pittsburgh, PA), 233--237.
• 25: Ari Korhonen , Lauri Malmi, Algorithm simulation with automatic assessment, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.160-163, July 11-13, 2000, Helsinki, Finland
• 26: Kostadinov, R. and Kumar, A. N. 2003. A tutor for learning encapsulation in C++ classes. In Proceedings of the ED-MEDIA 2003 World Conference on Educational Multimedia, Hypermedia and Telecommunications (Honolulu, HI), 1311--1314.
• 27: Markus Krebs , Tobias Lauer , Thomas Ottmann , Stephan Trahasch, Student-built algorithm visualizations for assessment: flexible generation, feedback and grading, Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education, June 27-29, 2005, Caparica, Portugal
• 28: Amruth N. Kumar, Generation of problems, answers, grade, and feedback---case study of a fully automated tutor, Journal on Educational Resources in Computing (JERIC), v.5 n.3, p.3-es, September 2005 [doi>10.1145/1163405.1163408]
• 29: Kumar, A. N. 2002. A tutor for using dynamic memory in C++. In Proceedings of the Frontiers in Education Conference (FIE 2002, Boston, MA), Session T4G.
• 30: Amruth N. Kumar, Results from the evaluation of the effectiveness of an online tutor on expression evaluation, Proceedings of the 36th SIGCSE technical symposium on Computer science education, February 23-27, 2005, St. Louis, Missouri, USA
• 31: Raymond Lister , Elizabeth S. Adams , Sue Fitzgerald , William Fone , John Hamer , Morten Lindholm , Robert McCartney , Jan Erik Moström , Kate Sanders , Otto Seppälä , Beth Simon , Lynda Thomas, A multi-national study of reading and tracing skills in novice programmers, ACM SIGCSE Bulletin, v.36 n.4, December 2004
• 32: Lauri Malmi , Ville Karavirta , Ari Korhonen , Jussi Nikander, Experiences on automatically assessed algorithm simulation exercises with different resubmission policies, Journal on Educational Resources in Computing (JERIC), v.5 n.3, p.7-es, September 2005 [doi>10.1145/1163405.1163412]
• 33: Merat, F. L. and Chung, D. 1997. World Wide Web approach to teaching microprocessors. In Proceedings of the FIE'97 Frontiers in Education Conference (Pittsburgh, PA), 838--841.
• 34: Pathak, S. and Brusilovsky, P. 2002. Assessing student programming knowledge with web-based dynamic parameterized quizzes. In Proceedings of the ED-MEDIA'2002---World Conference on Educational Multimedia, Hypermedia and Telecommunications (Denver, CO), 1548--1553.
• 35: Richard E. Pattis , Jim Roberts , Mark Stehlik, Karel the robot (2nd ed.): a gentle introduction to the art of programming, John Wiley & Sons, Inc., New York, NY, 1995
• 36: Harsh Shah , Amruth N. Kumar, A tutoring system for parameter passing in programming languages, ACM SIGCSE Bulletin, v.34 n.3, September 2002
• 37: Aaron P. Titus , L. W. Martin , Robert J. Beichner, Web-based testing in physics education: methods and opportunities, Computers in Physics, v.12 n.2, p.117-123, Mar/Apr 1998 [doi>10.1063/1.168628]
• 38: Weber, G. and Brusilovsky, P. 2001. ELM-ART: An adaptive versatile system for web-based instruction, Int. J. AI Edu. 12, 4, 351--384. Available at http://cbl.leeds.ac.uk/ijaied/abstracts/Vol_12/weber.html.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Distance learning

Additional Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Self-assessment

General Terms:
Human Factors, Languages, Performance

Keywords:
E-learning, assessment, classroom study, code execution, individualized exercises, introductory programming, parameterized questions