How shall we assess this?

ABSTRACT

Increased class sizes are forcing academics to reconsider approaches to setting and marking assessments for their students. Distributed and distance learning are creating some of the biggest changes. Some educators are embracing new technologies but others are more wary of what they do not know. In order to address this issue it is first necessary to investigate the types of assessment currently in use and the perceptions that are held by academics with and without experience of the new technologies that are becoming available.In this paper we present the findings of an international survey of Computer Science academics teaching a variety of topics within the discipline. The findings are split into two sections: a snapshot of current assessment practices and an analysis of respondents' perceptions of Computer Aided Assessment (CAA). Academics' opinions about the advantages and disadvantages of CAA are split in line with level of experience of using such techniques. Those with no experience of CAA suggest that it cannot be used to test higher-order learning outcomes and that the quality of the immediate feedback is poor; these negative opinions diminish as experience is gained.

REFERENCES

• 1: Ala-Mutka K, Computer-assisted Software Engineering Courses. Proceedings of IASTED International Conference Computers and Advanced Technology in Education, Cancun, 2002
• 2: Anderson W, Krathwohl D, A Taxonomy for Learning, Teaching and assessing. http://www.cours.polymtl.ca/plu6035/PDF/anderson.pdf, 2001
• 3: David Arnow , Oleg Barshay, On-line programming examinations using Web to teach, Proceedings of the 4th annual SIGCSE/SIGCUE ITiCSE conference on Innovation and technology in computer science education, p.21-24, June 27-30, 1999, Cracow, Poland
• 4: Arnow D, Barshay O, Web To Teach: an interactive focused programming exercise system, 29th Annual IEEE Frontiers in Education Conference, 1999
• 5: Ashworth P, Bannister P, Thorne P, Guilty in whose eyes? University students' perceptions of cheating and plagiarism in academic work and assessment, Studies in Higher Education, 22, 1997
• 6: Barnett DC, Dalton JC, Why college students cheat? Journal of College Student Personnel, 22, 1981
• 7: Bedny GZ, Seglin MH, Meister D, Activity theory: history, research and application, Theoretical Issues in Ergonomics Science, 1(2), 2000
• 8: Beevers E, Paterson JS, Automatic assessment of problemsolving skills in mathematics, Active Learning in Higher Education, 4(2), 2003
• 9: Blackboard, http://www.blackboard.com/
• 10: Bloom B, Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook I, Longman, New York, 1956
• 11: Duane Buck , David J. Stucki, Design early considered harmful: graduated exposure to complexity and structure based on levels of cognitive development, Proceedings of the thirty-first SIGCSE technical symposium on Computer science education, p.75-79, March 07-12, 2000, Austin, Texas, United States
• 12: Burstein J, Leacock C, Swartz R, Automated Evaluation of Essays and Short Answers, Proceedings of 5th Annual CAA Conference, Loughborough, 2001
• 13: Carneson J, Delpierre G, Masters K, Designing and Managing Multiple Choice Questions, http://www.le.ac.uk/cc/ltg/castle/resources/mcqman/mcqman01.html, 1999
• 14: Carpenter DD, Harding TS, Montgomery SM, Steneck N, P.A.C.E.S --- A study on academic integrity among engineering undergraduates (preliminary conclusions), ASEE Annual Conference & Exposition, Montreal, 2002
• 15: Chan CC, Tsui MS, Chan MYC, Applying the Structure of the Observed Learning Outcomes (SOLO) Taxonomy on Student's Learning Outcomes: an empirical study, Assessment & Evaluation in Higher Education, 27(6), 2002
• 16: Christie JR, Automated Essay Marking for both Style and Content, Proceedings of 3rd Annual CAA Conference, Loughborough, 1999
• 17: Clariana R, Wallace P, Paper-based versus computer-based assessment: key factors associated with the test mode effect, British Journal of Educational Technology, 33, 2002
• 18: Cook J, Leatherwood C, Oriogun P, Online Conferencing with Multimedia Students: Monitoring Gender Participation and Promoting Critical Debate, Proceedings of 2nd Annual LTSN-ICS Conference, London, 2001
• 19: Kevin Cox , David Clark, The use of formative quizzes for deep learning, Computers & Education, v.30 n.3-4, p.157-167, April/May 1998 [doi>10.1016/S0360-1315(97)00054-7]
• 20: Culwin F, MacLeod A, Lancaster T, Source Code Plagiarism in UK HE Computing Schools, Issues, Attitudes and Tools, Southbank University, London, Commissioned by JISC, 2001
• 21: Charlie Daly, RoboProf and an introductory computer programming course, Proceedings of the 4th annual SIGCSE/SIGCUE ITiCSE conference on Innovation and technology in computer science education, p.155-158, June 27-30, 1999, Cracow, Poland
• 22: Dalziel J, Gazzard S, Next generation computer assisted assessment software: the design and implementation of WebMCQ, Proceedings of 3rd Annual CAA Conference, Loughborough 1999
• 23: Davies P, Computer Aided Assessment MUST be more than multiple-choice tests for it to be academically credible? Proceedings of 5th Annual CAA Conference, Loughborough 2001
• 24: Davis H, Carr L, Cooke E, White S, Managing Diversity: Experiences Teaching Programming Principles in Proceedings of 2nd Annual LTSN-ICS Conference, London, 2001
• 25: Delandshire G, Implicit theories, unexamined assumptions and the status quo of educational assessment, Assessment in Education, 8(2), 2001
• 26: Dick M, Sheard J, Markham S, Is it OK to cheat? Proceedings of ITiCSE'01, Canterbury, 2001
• 27: Diekhoff GM, LaBeff EE, Clark RE, Williams LE, Francis B, and Haines VJ, College cheating: ten years later, Research in Higher Education, 37, 1996
• 28: Duke-Williams E, King T, Using Computer-Aided Assessment to Test Higher Level Learning Outcomes, Proceedings of 5th Annual CAA Conference, Loughborough, 2001
• 29: Dunn R, Beaudy J, Kalvan A, Survey of research on learning styles, Education Leadership, 46, 1989
• 30: John English , Phil Siviter, Experience with an automatically assessed course, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.168-171, July 11-13, 2000, Helsinki, Finland
• 31: John English, Experience with a computer-assisted formal programming examination, Proceedings of the 7th annual conference on Innovation and technology in computer science education, June 24-28, 2002, Aarhus, Denmark
• 32: Dave W. Farthing , Dave M. Jones , Duncan McPhee, Permutational multiple-choice questions: an objective and efficient alternative to essay-type examination questions, Proceedings of the 6th annual conference on the teaching of computing and the 3rd annual conference on Integrating technology into computer science education: Changing the delivery of computer science education, p.81-85, August 18-21, 1998, Dublin City Univ., Ireland
• 33: Farthing DW, McPhee D, Multiple choice for honours-level students? A statistical evaluation, Proceedings of 3rd Annual CAA Conference, Loughborough, 1999
• 34: Freeman M, McKenzie, SPARK, a confidential web-based template for self and peer assessment of student teamwork: benefits of evaluating across different subjects, British Journal of Educational Technology, 33, 2002
• 35: G. Frosini , B. Lazzerini , F. Marcelloni, Performing automatic exams, Computers & Education, v.31 n.3, p.281-300, Nov. 1998 [doi>10.1016/S0360-1315(98)00042-6]
• 36: Fuller M, Assessment for real in Virtual Learning Environments --- how far can we go? Proceedings of Interface --- Virtual Learning and Higher Education, Mansfield College, Oxford 2002
• 37: Ursula Fuller , John Slater , Gill Tardivel, Virtual seminars—real networked results? (poster), Proceedings of the 6th annual conference on the teaching of computing and the 3rd annual conference on Integrating technology into computer science education: Changing the delivery of computer science education, p.281, August 18-21, 1998, Dublin City Univ., Ireland
• 38: Dianne Hagan , Judy Sheard , Ian Macdonald, Monitoring and evaluating a redesigned first year programming course, Proceedings of the 2nd conference on Integrating technology into computer science education, p.37-39, June 01-05, 1997, Uppsala, Sweden
• 39: Hargreaves DJ, Student learning and assessment are inextricably linked, European Journal of Engineering Education 22(4), 1997
• 40: Hatzimoysis A, A Philosophy Primer in Virtual Seminars http://www.prsltsn.leeds.ac.uk/philosophy/events/hatzimoysis1.html, 2002
• 41: Colin Higgins , Pavlos Symeonidis , Athanasios Tsintsifas, The marking system for CourseMaster, Proceedings of the 7th annual conference on Innovation and technology in computer science education, June 24-28, 2002, Aarhus, Denmark
• 42: Hogarth S, Virtual Seminars in Psychology and Computinghttp://ctipsy.york.ac.uk/aster/resources/case_studies/reports/yk_6/yk_06.html, 2002
• 43: Gil Hoggarth , Mike Lockyer, An automated student diagram assessment system, Proceedings of the 6th annual conference on the teaching of computing and the 3rd annual conference on Integrating technology into computer science education: Changing the delivery of computer science education, p.122-124, August 18-21, 1998, Dublin City Univ., Ireland
• 44: Hunt N, Hughes J, Rowe G, Formative automated computer testing (FACT), British Journal of Educational Technology, 33, 2002
• 45: IMS http://www.imsglobal.org
• 46: David Jackson, A semi-automated approach to online assessment, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.164-167, July 11-13, 2000, Helsinki, Finland
• 47: David Jackson, A software system for grading student computer programs, Computers & Education, v.27 n.3-4, p.171-180, Dec. 1996 [doi>10.1016/S0360-1315(96)00025-5]
• 48: Janson S, Gender and the Information Society: A Socially Structured Silence, in Siefert M, Gerbner G, Fisher J (Eds.), The Information Gap: How Computers and Other Communication Technologies Affect the Social Distribution of Power, Oxford: Oxford University Press, 1989
• 49: Mike Joy , Michael Luck, Effective electronic marking for on-line assessment, Proceedings of the 6th annual conference on the teaching of computing and the 3rd annual conference on Integrating technology into computer science education: Changing the delivery of computer science education, p.134-138, August 18-21, 1998, Dublin City Univ., Ireland
• 50: JPLAG, http://www.jplag.de/
• 51: Kashy E, Tsai Y, Thoenssen M, Morrissey D, CAPA, An Integrated Computer Assisted Personal Assignment System, American Journal of Physics, 61(12), 1993
• 52: Ari Korhonen , Lauri Malmi, Algorithm simulation with automatic assessment, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.160-163, July 11-13, 2000, Helsinki, Finland
• 53: Amruth N. Kumar, Learning the interaction between pointers and scope in C++, Proceedings of the 6th annual conference on Innovation and technology in computer science education, p.45-48, June 2001, Canterbury, United Kingdom
• 54: Lee S, Development of instructional strategy of computer application software for group instruction in Computers and Education, 37(1), 2001
• 55: Le Heron J, Plagiarism, learning dishonest or just plain cheating: The context and countermeasures in information systems teaching, Australian Journal of Educational Technology, 17, 2001
• 56: Raymond Lister, Objectives and objective assessment in CS1, Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, p.292-296, February 2001, Charlotte, North Carolina, United States
• 57: Lauri Malmi , Ari Korhonen , Riku Saikkonen, Experiences in automatic assessment on mass courses and issues for designing virtual courses, Proceedings of the 7th annual conference on Innovation and technology in computer science education, June 24-28, 2002, Aarhus, Denmark
• 58: David V. Mason , Denise M. Woit, Integrating technology into computer science examinations, Proceedings of the twenty-ninth SIGCSE technical symposium on Computer science education, p.140-144, February 26-March 01, 1998, Atlanta, Georgia, United States
• 59: Mason O, Grove-Stephenson I, Automated free text marking with Paperless School, Proceedings of 6th Annual CAA Conference, Loughborough 2002
• 60: M. Dee Medley, On-line finals for CS1 and CS2, Proceedings of the 6th annual conference on the teaching of computing and the 3rd annual conference on Integrating technology into computer science education: Changing the delivery of computer science education, p.178-180, August 18-21, 1998, Dublin City Univ., Ireland
• 61: MOSS, http://www.cs.berkley.edu/~aiken/moss.html
• 62: Mulligan B, Pilot study on the impact of frequent computerized assessment on student work rates, Proceedings of 3rd Annual CAA Conference, Loughborough, 1999
• 63: Question Mark, http://www.questionmark.com/
• 64: Ross JL, Drysdale MTB, & Schulz RA, Cognitive learning styles and academic performance in two postsecondary computer courses. Journal of Research on Computing in Education, 33(4), 2001
• 65: Riku Saikkonen , Lauri Malmi , Ari Korhonen, Fully automatic assessment of programming exercises, Proceedings of the 6th annual conference on Innovation and technology in computer science education, p.133-136, June 2001, Canterbury, United Kingdom
• 66: Shimatani H, Kitao K, Computer Aided Instruction: A bibliography, http://ilc2.doshisha.ac.jp/users/kkitao/library/biblio/caibib.htm
• 67: C. Simon , Peter Summons, Automated testing of databases and spreadsheets - the long and the short of it, Proceedings of the Australasian conference on Computing education, p.215-219, December 2000, Melbourne, Australia [doi>10.1145/359369.359402]
• 68: Sloman J, Use of virtual seminars in Economic Principles, http://www.economics.ltsn.ac.uk/showcase/sloman_virtual.htm, 2002
• 69: Smailes J, Experiences of using Computer Aided Assessment within a Virtual Learning Environment, http://www.business.ltsn.ac.uk/events/BEST 2002/Papers/smailes.PDF, 2002
• 70: Solomon A, http://www-staff.it.uts.edu.au/~andrews/personal communication, 2003
• 71: M. Thelwall, Computer-based assessment: a versatile educational tool, Computers & Education, v.34 n.1, p.37-49, Jan. 2000 [doi>10.1016/S0360-1315(99)00037-8]
• 72: Thomas P, Price B, Petre M, Carswell L, Richards M, Experiments with Electronic Examinations over the Internet, Proceedings of 5th Annual CAA Conference, Loughborough 2001
• 73: Thomas P, Price B, Paine C, Richards M, Remote electronic examinations: student experiences, British Journal of Educational Technology, 33, 2002
• 74: Gareth Thorburn , Glenn Rowe, PASS: an automated system for program assessment, Computers & Education, v.29 n.4, p.195-206, Dec. 1997 [doi>10.1016/S0360-1315(97)00021-3]
• 75: Guglielmo Trentin, Computerized adaptive tests and formative assessment, Journal of Educational Multimedia and Hypermedia, v.6 n.2, p.201-220, 1997
• 76: Ward A, Using peer assessment assisted by ICT for programming assignments, Interactions, 5(1), 2001
• 77: WebCT, http://www.webct.com/
• 78: White S, Davis H, Creating large-scale test banks: a briefing for participative discussion of issues and agendas, Proceedings of 4th Annual CAA Conference, Loughborough 2000
• 79: Whittington D, Hunt H, Approaches to the computerized assessment of free text responses, Proceedings of 3rd Annual CAA Conference, Loughborough 1999
• 80: Wood D, Aspects of Teaching and Learning, in Light P, Sheldon S and Woodhead M (eds.) Learning to Think, Routledge, London, 1991

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.2 Computer and Information Science Education
          Subjects: Computer science education

General Terms:
Human Factors

Keywords:
assessment, computer aided assessment, plagiarism