Experiences with marmoset: designing and using an advanced submission and testing system for programming courses

ABSTRACT

We developed Marmoset, an automated submission and testing system, to explore techniques to provide improved feedback to both students and instructors as students work on programming assignments, and to collect data to perform detailed research on the development processes of students. To address the issue of feedback, Marmoset provides students with limited access to the results of the instructor's private test cases using a novel token-based incentive system. This both encourages students to start their work early and to think critically about their work. Because students submit early, instructors can monitor all students' progress on test cases, helping identify challenging or ambiguous test cases early in order to update the project specification or devote additional time in lecture or lab sessions to the difficult test cases.To study and better understand the development process of students, Marmoset can be configured to transparently capture snapshots to a central repository everytime students save their files. These detailed development histories offer a unique, detailed perspective of each student's progress on a programming assignment, from the first line of code written and saved all the way through the final edit before the final submission. This type of data has proven extremely valuable many uses, such as mining new bug patterns and evaluating existing bug-finding tools.In this paper, we describe our initial experiences using Marmoset in several introductory computer science courses, from the perspectives of both instructors and students. We also describe some initial research results from analyzing the student snapshot database.

REFERENCES

• 1: Stephen H. Edwards, Rethinking computer science education from a test-first perspective, Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 26-30, 2003, Anaheim, CA, USA [doi>10.1145/949344.949390]
• 2: Christopher C. Ellsworth , James B. Fenwick, Jr. , Barry L. Kurtz, The Quiver system, ACM SIGCSE Bulletin, v.36 n.1, March 2004
• 3: David Jackson , Michelle Usher, Grading student programs using ASSYST, Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education, p.335-339, February 27-March 01, 1997, San Jose, California, United States
• 4: Edward L. Jones, Grading student programs - a software testing approach, Proceedings of the fourteenth annual consortium on Small Colleges Southeastern conference, p.185-192, November 2000, Roanoke College, Salem, Virginia, United States
• 5: JUnit, testing resources for extreme programming. http://junit.org, 2004.
• 6: Y. Liu, E. Stroulia, K. Wong, and D. German. Using CVS historical information to understand how students develop software. In Proceedings of the International Workshop on Mining Software Repositories, Edinburgh, Scotland, May 2004.
• 7: Lauri Malmi , Ari Korhonen , Riku Saikkonen, Experiences in automatic assessment on mass courses and issues for designing virtual courses, Proceedings of the 7th annual conference on Innovation and technology in computer science education, June 24-28, 2002, Aarhus, Denmark
• 8: Andrew Oram , Steve Talbot, Managing Projects with Make, O'Reilly & Associates, Inc., Sebastopol, CA, 1992
• 9: Kenneth A. Reek, A software infrastructure to support introductory computer science courses, Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education, p.125-129, February 15-17, 1996, Philadelphia, Pennsylvania, United States
• 10: Riku Saikkonen , Lauri Malmi , Ari Korhonen, Fully automatic assessment of programming exercises, Proceedings of the 6th annual conference on Innovation and technology in computer science education, p.133-136, June 2001, Canterbury, United Kingdom
• 11: Nghi Truong , Paul Roe , Peter Bancroft, Static analysis of students' Java programs, Proceedings of the sixth conference on Australasian computing education, p.317-325, January 01, 2004, Dunedin, New Zealand
• 12: Andreas Zeller, Making students read and review code, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.89-92, July 11-13, 2000, Helsinki, Finland

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

General Terms:
Experimentation, Human Factors, Measurement

Keywords:
code snapshots, project submission, testing