Evaluating assessments of novice programming environments

ABSTRACT

Educators have developed a myriad of tools to help novices learn to program. Unfortunately, too little is known about the educational impact of these environments, or even how to assess this impact. In this paper we summarize a representative collection of the assessments of novice programming environments, present a rubric for evaluating the quality of such assessments, and demonstrate the application of our rubric to the summarized works. The intent is that such an evaluative framework will help inform future efforts in assessing novice programming environments.

REFERENCES

• 1: Alice, 2005. Online. Internet. Apr. 13, 2005. Available WWW: <http://www.alice.org>
• 2: Byron Weber Becker, Teaching CS1 with karel the robot in Java, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.50-54, February 2001, Charlotte, North Carolina, United States
• 3: Stephen A. Bloch, Scheme and Java in the first year, Proceedings of the fifth annual CCSC northeastern conference on The journal of computing in small colleges, p.157-165, April 2000, Ramapo College of New Jersey, Mahwah, New Jersey, United States
• 4: Bloom, B., 1956. Taxonomy of educational objectives: The classification of educational goals. Longman Publishers, New York.
• 5: BlueJ, 2005. Online. Internet. Apr. 13, 2005. Available WWW: http://www.bluej.org
• 6: Richard Borge , Annita Fjuk , Arne-Kristian Groven, Using Karel J Collaboratively to Facilitate Object-Oriented Learning, Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT'04), p.580-584, August 30-September 01, 2004
• 7: Kim B. Bruce , Andrea Danyluk, Event-driven programming facilitates learning standard programming concepts, Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, October 24-28, 2004, Vancouver, BC, CANADA [doi>10.1145/1028664.1028704]
• 8: Duane Buck , David J. Stucki, JKarelRobot: a case study in supporting levels of cognitive development in the computer science curriculum, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.16-20, February 2001, Charlotte, North Carolina, United States
• 9: Martin C. Carlisle , Terry A. Wilson , Jeffrey W. Humphries , Steven M. Hadfield, RAPTOR: a visual programming environment for teaching algorithmic problem solving, Proceedings of the 36th SIGCSE technical symposium on Computer science education, February 23-27, 2005, St. Louis, Missouri, USA
• 10: Cross, J., T. Hendrix, and L. Barowski, 2002. Using the debugger as an integral part of teaching CS1. In Proceedings of the 32nd ASEE/IEEE Frontiers in Education Conference, Boston, Massachusetts, available on CDROM ISBN 0-7803-7445-2.
• 11: Fagin, B., 2003. Ada/Mindstorms 3.0: A Computational Environment for Introductory Robotics and Programming. Robotics & Automation Magazine, IEEE. Vol.10, Iss.2, 19--24.
• 12: Barry S. Fagin , Laurence Merkle, Quantitative analysis of the effects of robots on introductory Computer Science education, Journal on Educational Resources in Computing (JERIC), v.2 n.4, p.2-es, December 2002 [doi>10.1145/949257.949259]
• 13: Sally Fincher , Marian Petre, Computer Science Education Research, 2004
• 14: Andrea Forte , Mark Guzdial, Computers for Communication, Not Calculation: Media as a Motivation and Context for Learning, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 4, p.40096.1, January 05-08, 2004
• 15: Goldman, K. An interactive environment for beginning Java programmers. Science of Computer Programming. Vol. 53, Iss. 1, 3--24. http://dx.doi.org/10.1016/j.scico.2004.02.002
• 16: Gross, P., and K. Powers, 2005. A Meta-Study of Novice Programming Environments. Tufts University, Department of Computer Science, Technical Report. In preparation.
• 17: Guzdial, M., 2004. Programming environments for novices. In Computer Science Education Research. S. Fincher and M. Petre (Eds.). Swets and Zeitlinger. Chapter 3.
• 18: Hundhausen C., S. Douglas and J. Stasko, 2002. A meta-study of algorithm visualization effectiveness. Journal of Visual Languages & Computing. Vol. 13, Iss. 3, 259--290.
• 19: Jeliot 3, 2005. Online. Internet. Apr. 13, 2005. Available WWW: http://www.cs.joensuu.fi/jeliot/index.php
• 20: Caitlin Kelleher , Randy Pausch, Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers, ACM Computing Surveys (CSUR), v.37 n.2, p.83-137, June 2005 [doi>10.1145/1089733.1089734]
• 21: Lego Mindstorms, 2005. Online. Internet. Apr. 13, 2005. <http://www.legomindstorms.com/>
• 22: Ronit Ben-Bassat Levy , Mordechai Ben-Ari , Pekka A. Uronen, The Jeliot 2000 program animation system, Computers & Education, v.40 n.1, p.1-15, January 2003 [doi>10.1016/S0360-1315(02)00076-3]
• 23: Long, A. and M. Godfrey, 2004. An Evaluation Tool to Assess the Quality of Qualitative Research Studies. International Journal of Social Research Methodology. Vol.7, Iss.2, 181--196.
• 24: Javier López , Niko Myller , Erkki Sutinen, Sorting out sorting through concretization with robotics, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989929]
• 25: Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer , Lynda Thomas , Ian Utting , Tadeusz Wilusz, A multi-national, multi-institutional study of assessment of programming skills of first-year CS students, Working group reports from ITiCSE on Innovation and technology in computer science education, December 01, 2001, Canterbury, UK [doi>10.1145/572133.572137]
• 26: Barbara Moskal , Deborah Lurie , Stephen Cooper, Evaluating the effectiveness of a new instructional approach, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 27: Thomas Naps , Stephen Cooper , Boris Koldehofe , Charles Leska , Guido Rößling , Wanda Dann , Ari Korhonen , Lauri Malmi , Jarmo Rantakokko , Rockford J. Ross , Jay Anderson , Rudolf Fleischer , Marja Kuittinen , Myles McNally, Evaluating the educational impact of visualization, Working group reports from ITiCSE on Innovation and technology in computer science education, p.124-136, June 30-July 02, 2003, Thessaloniki, Greece
• 28: Keith J. O'Hara , Jennifer S. Kay, Investigating open source software and educational robotics, Journal of Computing Sciences in Colleges, v.18 n.3, p.8-16, February 2003
• 29: Seymour Papert, Mindstorms: Children, computers, and powerful ideas, Basic Books, Inc., New York, NY, 1980
• 30: Richard E. Pattis , Jim Roberts , Mark Stehlik, Karel the robot (2nd ed.): a gentle introduction to the art of programming, John Wiley & Sons, Inc., New York, NY, 1995
• 31: Tamar Paz , Tami Lapidot, Emergence of automated assignment conceptions in a functional programming course, Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education, June 28-30, 2004, Leeds, United Kingdom
• 32: Pellegrino, J., N. Chudowski, and R. Glaser, 2001. Knowing What Students Know; The Science and Design of Educational Assessment. Washington DC: National Academy Press.
• 33: Noa Ragonis , Mordechai Ben-Ari, On understanding the statics and dynamics of object-oriented programs, Proceedings of the 36th SIGCSE technical symposium on Computer science education, February 23-27, 2005, St. Louis, Missouri, USA
• 34: RAPTOR, 2005. Online. Internet. Apr. 13, 2005. Available WWW: atlas.usafa.af.mil/dfcs/bios/mcc_html/raptor.html
• 35: Richard Rasala , Jeff Raab , Viera K. Proulx, Java power tools: model software for teaching object-oriented design, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.297-301, February 2001, Charlotte, North Carolina, United States
• 36: Lauren Rich , Heather Perry , Mark Guzdial, A CS1 course designed to address interests of women, Proceedings of the 35th SIGCSE technical symposium on Computer science education, March 03-07, 2004, Norfolk, Virginia, USA
• 37: Sajaniemi, J. and M. Kuittinen, 2005. An Experiment on Using Roles of Variables in Teaching Introductory Programming. Journal of Computer Science Education. Vol. 15, Iss. 1, 59--82.
• 38: Sanders, D. and B. Dorn, 2004. Object-Oriented Programming with Jeroo in the Information Technology Classroom <http://www.cc.gatech.edu/%7Edorn/papers/ISECON04.pdf>. Proceedings of ISECON 2004. CD-ROM.
• 39: Van Haaster K., 2003. Introductory Programming in an OO Environment: An Evaluation of a Pedagogical Tool, Honours Thesis, Monash University. Available WWW: http://www.csse.monash.edu.au/ khaaster/thesis3.zip
• 40: Van Haaster, K. and D. Hagan, 2004. Teaching and Learning with BlueJ: an Evaluation of a Pedagogical Tool. Information Science and Information Technology Education Joint Conference, Rockhampton, QLD, Australia.
• 41: Wang, E., 2001. Teaching freshmen design, creativity, and programming with Legos and Labview. In Proceedings of the 31st ASEE/IEEE Frontiers in Education Conference, Reno, Nevada, available on CDROM ISBN: 0-7803-6671-9.
• 42: Williams, A., 2003. The qualitative impact on using Lego Mindstorms robots to teach computer engineering. IEEE Transactions on Education. Vol. 46, Iss. 1, 206.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

General Terms:
Design, Experimentation, Measurement

Keywords:
assessment, evaluation, novice programmers, novice programming environments