Multi-institutional, multi-national studies in CSEd Research: some design considerations and trade-offs

ABSTRACT

One indication of the maturation of Computer Science Education as a research-based discipline is the recent emergence of several large-scale studies spanning multiple institutions. This paper examines a "family" of these multi-institutional, multi-national studies, detailing core elements and points of difference in both study design and the organization of the research team, and highlighting the costs and benefits associated with the different approaches.

REFERENCES

• 1: Porter, A.C. and A. Gamoran, eds. Methodological Advances in Cross-National Surveys of Educational Achievement. 2002, National Academy Press: Washington DC.
• 2: Michael McCracken , Vicki Almstrum , Danny Diaz , Mark Guzdial , Dianne Hagan , Yifat Ben-David Kolikant , Cary Laxer , Lynda Thomas , Ian Utting , Tadeusz Wilusz, A multi-national, multi-institutional study of assessment of programming skills of first-year CS students, ACM SIGCSE Bulletin, v.33 n.4, December 2001 [doi>10.1145/572139.572181]
• 3: Petre, M., et al., "My criterion is: Is it a Boolean?": A card-sort elicitation of students' knowledge of programming constructs. 2003, University of Kent: Canterbury. p. 37.
• 4: Fincher, S., et al., Cause for alarm? A multi-national, multi-institutional study of student-generated software designs. 2004, Computing Laboratory, University of Kent: Canterbury.
• 5: Fincher, S., et al., Programmed to succeed? A multi-national, multi-institutional study of introductory programming courses. 2005, Computing Laboratory, University of Kent: Canterbury.
• 6: Raymond Lister , Elizabeth S. Adams , Sue Fitzgerald , William Fone , John Hamer , Morten Lindholm , Robert McCartney , Jan Erik Moström , Kate Sanders , Otto Seppälä , Beth Simon , Lynda Thomas, A multi-national study of reading and tracing skills in novice programmers, ACM SIGCSE Bulletin, v.36 n.4, December 2004
• 7: McCauley, R., et al., What do successful computer science students know? An integrative analysis using card sort measures and content analysis to evaluate graduating students' knowledge of programming concepts. Expert Systems, 2005. 22(3): p. 147--159.
• 8: Gilford, D.M., ed. A Collaborative Agenda for Improving International Comparative Studies in Education. 1993, National Academy Press: Washington DC.
• 9: Mathiassen, L., Collaborative practice research. Information Technology and People, 2002. 15(4): p. 321--245.
• 10: McKay, J. and P. Marshall, The dual imperatives of action research. Information Technology and People, 2001. 14(1): p. 46--59.
• 11: Taylor, P., Institutional Change in Uncertain Times: Lone ranging is not enough. Studies in Higher Education, 1998. 23(3): p. 269--279.
• 12: Ainslie Ellis , Linda Carswell , Andrew Bernat , Daniel Deveaux , Patrice Frison , Veijo Meisalo , Jeanine Meyer , Urban Nulden , Joze Rugelj , Jorma Tarhio, Resources, tools, and techniques for problem based learning in computing, Working Group reports of the 3rd annual SIGCSE/SIGCUE ITiCSE conference on Integrating technology into computer science education, p.41-56, December 1998, Dublin, Ireland [doi>10.1145/316572.358296]
• 13: Pamela B. Lawhead , Michaele E. Duncan , Constance G. Bland , Michael Goldweber , Madeleine Schep , David J. Barnes , Ralph G. Hollingsworth, A road map for teaching introductory programming using LEGO© mindstorms robots, ACM SIGCSE Bulletin, v.35 n.2, June 2003
• 14: Janet Carter , Kirsti Ala-Mutka , Ursula Fuller , Martin Dick , John English , William Fone , Judy Sheard, How shall we assess this?, ACM SIGCSE Bulletin, v.35 n.4, December 2003
• 15: Vicki L. Almstrum , Nell Dale , Anders Berglund , Mary Granger , Joyce Currie Little , Diane M. Miller , Marian Petre , Paul Schragger , Fred Springsteel, Evaluation: turning technology from toy to tool: report of the working group on evaluation, ACM SIGCSE Bulletin, v.28 n.SI, p.201-217, 1996
• 16: Bootstrapping, Bootstrapping Research in Computer Science Education. 2002. http://depts.washington.edu/bootstrp/
• 17: Laurie Murphy , Renée McCauley , Suzanne Westbrook , Timothy Fossum , Susan Haller , Briana B. Morrison , Brad Richards , Kate Sanders , Carol Zander , Ruth E. Anderson, A multi-institutional investigation of computer science seniors' knowledge of programming concepts, Proceedings of the 36th SIGCSE technical symposium on Computer science education, February 23-27, 2005, St. Louis, Missouri, USA
• 18: Whalley, J., Report on the BRACElet Workshop. 2004, Auckland University of Technology: Auckland.
• 19: Experiment-Kits, Experiment Kit Page. 2005. http://www.cs.kent.ac.uk/people/staff/saf/experiment-kits/index.html
• 20: Sanders, K., et al., A multi-institutional, multinational study of programming concepts using card sort data. Expert Systems, 2005. 22(3): p. 121--128.
• 21: McCartney, R., et al. Questions, Annotations, and Institutions: observations from a study of novice programmers. in Kolin Kolistelut - Koli Calling. 2004. Koli, Finland.
• 22: BRACE, Building Research in Australasian Computing Education. 2004. http://www.cs.otago.ac.nz/brace/
• 23: de Raadt, M., et al. Approaches to Learning in Computer Programming Students, and its Effect on Success. in Annual International Conference of the Higher Education Research and Development Society of Australis (HERDSA 2005). 2005. Sydney, Australia.
• 24: Deibel, K., R. Anderson, and R. Anderson, Using edit distance to analyze card sorts. Expert Systems, 2005. 22(3).
• 25: Fincher, S. and J. Tenenberg, Making sense of card-sorting data. Expert Systems, 2005. 22(3).
• 26: Fossum, T. and S. Haller, Measuring card sort orthogonality. Expert Systems, 2005. 22(3).
• 27: Tenenberg, J., et al., Students designing software: a multi-national, multi-institutional study. Informatics in Education, 2005. 4(1): p. 143--162.
• 28: Anderson, L.W., et al., eds. A taxonomy for learning, teaching, and assessing: a revision of Bloom's Taxonomy of educational objectives. Complete ed. 2001, Longman: New York. xxix, 352.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

Additional Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Curriculum

General Terms:
Documentation, Experimentation, Management, Measurement

Keywords:
education, empirical, multi-institutional, research