Software repository mining with Marmoset: an automated programming project snapshot and testing system

ABSTRACT

Most computer science educators hold strong opinions about the "right" approach to teaching introductory level programming. Unfortunately, we have comparatively little hard evidence about the effectiveness of these various approaches because we generally lack the infrastructure to obtain sufficiently detailed data about novices' programming habits.To gain insight into students' programming habits, we developed Marmoset, a project snapshot and submission system. Like existing project submission systems, Marmoset allows students to submit versions of their projects to a central server, which automatically tests them and records the results. Unlike existing systems, Marmoset also collects finegrained code snapshots as students work on projects: each time a student saves her work, it is automatically committed to a CVS repository.We believe the data collected by Marmoset will be a rich source of insight about learning to program and software evolution in general. To validate the effectiveness of our tool, we performed an experiment which found a statistically significant correlation between warnings reported by a static analysis tool and failed unit tests.To make fine-grained code evolution data more useful, we present a data schema which allows a variety of useful queries to be more easily formulated and answered.

REFERENCES

• 1: CheckStyle. http://checkstyle.sourceforge.net, 2005.
• 2: CVS. http://www.cvshome.org. 2004.
• 3: Eclipse.org main page. http://www.eclipse.org, 2004.
• 4: Stephen H. Edwards, Rethinking computer science education from a test-first perspective, Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 26-30, 2003, Anaheim, CA, USA [doi>10.1145/949344.949390]
• 5: David Hovemeyer , William Pugh, Finding bugs is easy, Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, October 24-28, 2004, Vancouver, BC, CANADA [doi>10.1145/1028664.1028717]
• 6: David Jackson , Michelle Usher, Grading student programs using ASSYST, Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education, p.335-339, February 27-March 01, 1997, San Jose, California, United States
• 7: Edward L. Jones, Grading student programs - a software testing approach, Proceedings of the fourteenth annual consortium on Small Colleges Southeastern conference, p.185-192, November 2000, Roanoke College, Salem, Virginia, United States
• 8: JUnit, testing resources for extreme programming. http://junit.org, 2004.
• 9: Y. Liu, E. Stroulia, K. Wong, and D. German. Using CVS historical information to understand how students develop software. In Proceedings of the International Workshop on Mining Software Repositories, Edinburgh, Scotland, May 2004.
• 10: PMD. http://pmd.sourceforge.net, 2005.
• 11: Kenneth A. Reek, A software infrastructure to support introductory computer science courses, Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education, p.125-129, February 15-17, 1996, Philadelphia, Pennsylvania, United States
• 12: K. A. Schneider, C. Gutwin, R. Penner, and D. Paquette. Mining a software developer's local interaction history. In Proceedings of the International Workshop on Mining Software Repositories, Edinburgh, Scotland, May 2004.
• 13: Jaime Spacco , David Hovemeyer , William Pugh, An Eclipse-based course project snapshot and submission system, Proceedings of the 2004 OOPSLA workshop on eclipse technology eXchange, p.52-56, October 24-24, 2004, Vancouver, British Columbia, Canada [doi>10.1145/1066129.1066140]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Data mining

General Terms:
Design, Experimentation, Measurement, Performance