Partnership reviewing: a cooperative approach for peer review of complex educational resources

ABSTRACT

Review of digital educational resources, such as course modules, simulations, and data analysis tools, can differ from review of scholarly articles, in the heterogeneity and complexity of the resources themselves. The Partnership Review Model, as demonstrated in two cases, appears to promote cooperative interactions between distributed resource reviewers, enabling reviewers to effectively divide up the task of reviewing complex resources with little explicit coordination. The shared structural outline of the resource made visible in the review environment enables participants to monitor other reviewers' actions and to thus target their efforts accordingly. This reviewing approach may be effective in educational digital libraries that depend on community volunteers for most of their reviewing.

REFERENCES

• 1: AAAS Benchmarks for Science Literacy. American Association for the Advancement of Science (AAAS), Washington DC, 1993
• 2: Archer, J. Electronic publishing in the humanities and social sciences, http://ahdp.lib.ucalgary.ca/hssfc, 2000
• 3: Arms, W.Y. Digital Libraries. MIT Press, Cambridge, 2000
• 4: Axelrod, R. The Evolution of Cooperation. Basic Books, New York, 1984
• 5: Buckingham Shum, S., and Sumner, T. JIME: An Interactive Journal for Interactive Media. First Monday, 6 (2 February), 2001
• 6: Elizabeth F. Churchill , Jonathan Trevor , Sara Bly , Les Nelson , Davor Cubranic, Anchored conversations: chatting in the context of a document, Proceedings of the SIGCHI conference on Human factors in computing systems, p.454-461, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332475]
• 7: Cicchetti, D. Reliability of reviews for the American Psychologist. Am. Psychologist, 35. 300--303, 1980
• 8: Cicchetti, D. The reliabilty of peer review for manuscript and grant submissions. Behavioral and Brain Sciences, 14. 119--186, 1991
• 9: Cicchetti, D. and Conn, H. A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Jnl. of Biology and Medicine, 45. 373--383, 1976
• 10: Alistair Cockburn, Writing Effective Use Cases, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2000
• 11: Dillenbourg, P., Baker, M., Blaye, A. and O'Malley, C. The Evolution of Research on Collaborative Learning, in Reimann, P. and Spada, H. eds. Learning in humans and machines. Towards an interdisciplinary learning science, Pergamon, London, 1995
• 12: Fischer, G. The Importance of Models in Making Complex Systems Comprehensible, in Tauber, M. and Ackermann, D. eds. Mental Models and Human-Computer Interaction 2, Elsevier Science Publishers B. V., North Holland, 1991
• 13: Fjällbrant, N. Scholarly communication: Historical development and new possibilities, http://educate.lib.chalmers.se/IATUL/proceedcontents/fullpaper/nfpaper.html,, 1997
• 14: Hargens, L. and Herting, J. A new approach to referees' assessments of manuscripts. Social Science Research, 19. 1--16, 1990
• 15: Harnad, S. Peer commentary on Peer Review. Behavioral and Brain Sciences, 5 (2). 185--186, 1982
• 16: Harnad, S. Scholarly skywriting and the prepublication continuum of scientific inquiry. Psychological Science, 1. 342--343, 1990
• 17: James Hollan , Edwin Hutchins , David Kirsh, Distributed cognition: toward a new foundation for human-computer interaction research, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.2, p.174-196, June 2000 [doi>10.1145/353485.353487]
• 18: Hutchins, E. Cognition in the Wild. MIT Press, Cambridge, MA, 1996
• 19: Kastens, K.A. How to identify the "best" resources for the reviewed collection of the Digital Library for Earth System Education. Column "Another Node on the interNet," Computers & Geosciences, 27 (3). 375--378, 2001
• 20: Michael Khoo, Community design of DLESE's collections review policy: a technological frames analysis, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.157-164, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379474]
• 21: Michael Khoo, Ethnography, Evaluation, and Design as Integrated Strategies: A Case Study from WES, Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries, p.263-274, September 04-09, 2001
• 22: Kuhn, T. The structure of scientific revolutions. University of Chicago Press, Chicago, 1970
• 23: Manduca, C.A. and Mogk, D.W. The Digital Library for Earth System Education: A Community Plan. Final Report from the University of Oklahoma to the NSF, Grant 99-06648, 2000, 44
• 24: Mary Marlino , Tamara Sumner , David Fulker , Cathryn Manduca , David Mogk, The digital library for earth system education: building community, building the library, Communications of the ACM, v.44 n.5, p.80-81, May 2001 [doi>10.1145/374308.374356]
• 25: McKie, D. The scientific periodical from 1665 to 1798. Philosophical Magazine, Commemoration Issue. 122--132, 1948
• 26: Wanda J. Orlikowski , Debra C. Gash, Technological frames: making sense of information technology in organizations, ACM Transactions on Information Systems (TOIS), v.12 n.2, p.174-207, April 1994 [doi>10.1145/196734.196745]
• 27: Peters, D. and Ceci, S. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5 (2). 187--195, 1982
• 28: Psycoloquy. http://www.princeton.edu/~harnad/psyc.html
• 29: Putnam, R.D. Bowling alone: America's Declining Social Capital. Journal of Democracy, 6. 65--78, 1995
• 30: Tracy Riggs , Robert Wilensky, An algorithm for automated rating of reviewers, Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, p.381-387, January 2001, Roanoke, Virginia, United States [doi>10.1145/379437.379731]
• 31: Siemens, R. The credibility of electronic publishing, http://web.mala.bc.ca/hssfc/Final/Credibility.htm,, 2001
• 32: Speck, B. Publication peer review: An annotated bibliography. Greenwood, Westport, CN, 1993
• 33: Cheri Speier , Jonathan Palmer , Daniel Wren , Susan Hahn, Faculty perceptions of electronic journals as scholarly communication: a question of prestige and legitimacy, Journal of the American Society for Information Science, v.50 n.6, p.537-543, May 1, 1999 [doi>10.1002/(SICI)1097-4571(1999)50:6<537::AID-ASI9>3.3.CO;2-Y]
• 34: Tamara Sumner , Simon Buckingham Shum, From documents to discourse: shifting conceptions of scholarly publishing, Proceedings of the SIGCHI conference on Human factors in computing systems, p.95-102, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274659]
• 35: Sumner, T., Buckingham Shum, S., Wright, M., Bonnardel, N., Piolat, A. and Chevalier, A., Redisigning the peer review process: A developmental theory-in-action, in COOP 2000: Fourth International Conference on the Design of Cooperative Systems, (Sophia Antipolis, France, 2000)
• 36: Weatherley, J. Adapting educational resources for collaborative on-line peer review, MS thesis, Dept. of Computer Science, University of Colorado, Boulder, 2001, 87 pages
• 37: Zuckerman, H. and Merton, R. Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, IX (1). 66--100, 1971

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection; Standards
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Collaborative computing; Asynchronous interaction

General Terms:
Design, Experimentation, Human Factors, Management

Keywords:
computer-mediated communication, distributed cognition, educational digital libraries, peer review, scholarly publishing, second order interactional effects