Evaluating computer-supported cooperative work

ABSTRACT

Evaluating distributed CSCW applications is a difficult endeavor. Frameworks and methodologies for structuring this type of evaluation have become a central concern for CSCW researchers. In this paper we describe the problems involved in evaluating remote collaborations, and we review some of the more prominent conceptual frameworks of group interaction that have driven CSCW evaluation in the past. A multifaceted evaluation framework is presented that approaches the problem from the relationships underlying joint awareness, communication, collaboration, coordination, and work coupling. Finally, recommendations for carrying out multifaceted evaluations of remote interaction are provided.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Arrow, H., McGrath, J. E., and Berdahl, J. L. (2000). Small Groups as Complex Systems: Formation, Coordination, Development, and Adaptation. Thousand Oaks, CA: Sage.
	2	Uwe M. Borghoff , J. H. Schlichter, Computer-Supported Cooperative Work: Introduction to Distributed Applications, Springer-Verlag New York, Inc., Secaucus, NJ, 2000
	3	Frederick P. Brooks, Jr., The mythical man-month (anniversary ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
	4	John M. Carroll , Dennis C. Neale , Philip L. Isenhour , Mary Beth Rosson , D. Scott McCrickard, Notification and awareness: synchronizing task-oriented collaborative activity, International Journal of Human-Computer Studies, v.58 n.5, p.605-632, May 2003 [doi>10.1016/S1071-5819(03)00024-7]
	5	John M. Carroll , George Chin , Mary Beth Rosson , Dennis C. Neale, The development of cooperation: five years of participatory design in the virtual school, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.239-251, August 17-19, 2000, New York City, New York, United States [doi>10.1145/347642.347731]
	6	Clark, H. H. (1996). Using Language. New York: Cambridge University Press.
	7	Gregorio Convertino , Dennis C. Neale , Laurian Hobby , John M. Carroll , Mary Beth Rosson, A laboratory method for studying activity awareness, Proceedings of the third Nordic conference on Human-computer interaction, p.313-322, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028063]
	8	Laurie Damianos , Lynette Hirschman , Robyn Kozierok , Jeffrey Kurtz , Andrew Greenberg , Kimberley Walls , Sharon Laskowski , Jean Scholtz, Evaluation for collaborative systems, ACM Computing Surveys (CSUR), v.31 n.2es, June 1999 [doi>10.1145/323216.323362]
	9	Paul Dourish , Victoria Bellotti, Awareness and coordination in shared workspaces, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.107-114, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143468]
	10	Craig H. Ganoe , Jacob P. Somervell , Dennis C. Neale , Philip L. Isenhour , John M. Carroll , Mary Beth Rosson , D. Scott McCrickard, Classroom BRIDGE: using collaborative public and desktop timelines to support activity awareness, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.21-30, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964699]
	11	Greene, J. C., Caracelli, V. J., and Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255--274.
	12	Grudin, J. (1988). Why groupware applications fail: Problems in design and evaluation. Office: Technology and People, 4(3), 245--264.
	13	Jonathan Grudin, Group dynamics and ubiquitous computing, Communications of the ACM, v.45 n.12, December 2002 [doi>10.1145/585597.585618]
	14	Hare, A. P. (1992). Groups, teams, and social interaction: Theories and applications. New York: Praeger.
	15	Mark Hartswood , Rob Procter, Design guidelines for dealing with breakdowns and repairs in collaborative work settings, International Journal of Human-Computer Studies, v.53 n.1, p.91-120, July 2000 [doi>10.1006/ijhc.2000.0374]
	16	Helms, J., Neale, D. C., and Carroll, J. M. (2000). Data logging: Higher-level capturing and multi-level abstracting of user activities. In Proceedings of the 40th Annual Meeting of the Human Factors and Ergonomics Society (pp. 303--306). Santa Monica, CA: Human Factors and Ergonomics Society.
	17	Hendrick, H. W., and Kleiner, B. M. (2000). Macroergonomics: An introduction to work system design. (Vol. 2). Santa Monica, CA: Human Factors and Ergonomics Society.
	18	Humphries, W., Neale, D. C., McCrickard, D. S., and Carroll, J. M. (2004). Laboratory Simulation Methods for Studying Complex Collaborative Tasks. In Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society. Human Factors and Ergonomics Society (to appear). Santa Monica, CA.
	19	Isenhour, P. L., Carroll, J. M., Neale, D. C., Rosson, M. B., and Dunlap, D. R. (2000). The virtual school: An integrated collaborative environment for the classroom. Educational Technology and Society, Special Issue on "On-Line Collaborative Learning Environments", 3(3), http://ifets.ieee.org/periodical/.
	20	Klein, G. (2001). Features of team coordination. In M. McNeese, E. Salas, and M. Endsley (Eds.), New Trends in Cooperative Activities: Understanding System Dynamics in Complex Environments (pp. 68--95). Santa Monica, CA: Human Factors and Ergonomics Society.
	21	Thomas W. Malone , Kevin Crowston, The interdisciplinary study of coordination, ACM Computing Surveys (CSUR), v.26 n.1, p.87-119, March 1994 [doi>10.1145/174666.174668]
	22	McGrath, J. E. (1984). Groups: Interaction and performance. Englewoods Cliffs, NJ: Prentice-Hall.
	23	Joseph E. McGrath, Time matters in groups, Intellectual teamwork: social and technological foundations of cooperative work, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1990
	24	Joseph E. McGrath, Methodology matters: doing research in the behavioral and social sciences, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
	25	Miles, M. B., and Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.
	26	Bonnie A. Nardi, Context and consciousness: activity theory and human-computer interaction, Massachusetts Institute of Technology, Cambridge, MA, 1995
	27	Neale, D. C., and Carroll, J. M. (1999). Multi-faceted evaluation for complex, distributed activities. In Proceedings of CSCL'99 Computer Supported Cooperative Learning (pp. 425--433). Mahwah, New Jersy: Lawrence Erlbaum.
	28	Neale, D. C., Dunlap, D. R., Isenhour, P., and Carroll, J. M. (2000). Collaborative critical incident development. In Proceedings of the 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 598--601). Santa Monica, CA: Human Factors and Ergonomics Society.
	29	Olson, G. M., and Olson, J. S. (1997). Research on computer supported cooperative work. In M. Helander, T. K. Landauer, and P. Prablu (Eds.), Handbook of Human-Computer Interaction (pp. 1433--1456). Amsterdam, The Netherlands: Elsevier Science.
	30	Olson, G. M., and Olson, J. S. (2000). Distance matters. Human-Computer Interaction, 15, 139--178.
	31	Olson, G. M., and Olson, J. S. (2001). Technology support for collaborative workgroups. In G. M. Olson, T. W. Malone, and J. B. Smith (Eds.), Coordination Theory and Collaboration Technology (pp. 559--584). Mahwah, N. J.: Lawrence Erlbaum.
	32	Pedhazur, E. J. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, New Jersey: Lawrence Erlbaum.
	33	David Pinelle , Carl Gutwin, A Review of Groupware Evaluations, Proceedings of the 9th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, p.86-91, June 04-16, 2000
	34	David Pinelle , Carl Gutwin , Saul Greenberg, Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.4, p.281-311, December 2003 [doi>10.1145/966930.966932]
	35	Pinsonneault, A., and Kraemer, K. L. (1989). The impact of technological support on groups: An assessment of the empirical research. Decision Support Systems, 5(2), 197--211.
	36	Plowman, L., Rogers, Y., and Ramage, M. (1995). What are workplace studies for?. In Proceedings of ECSCW '95 European Conference on Computer-Supported Cooperative Work (pp. 309--324). Dordrecht: Kluwer.
	37	Ross, S., Ramage, M., and Rogers, Y. (1995). PETRA: Participatory evaluation through redesign and analysis. Interacting with Computers, 7(4), 335--360.
	38	Ruhleder, K., and Jordan, B. (1998). Video-based interaction analysis (VBIA) in distributed settings: A tool for analyzing multiple-site, technology-supported interactions. In Proceeding of the Participatory Design Conference (pp. 195--196). Seattle, WA.
	39	Tony Salvador , Jean Scholtz , James Larson, The Denver model for groupware design, ACM SIGCHI Bulletin, v.28 n.1, p.52-58, Jan. 1996 [doi>10.1145/249170.249185]
	40	Lee Sproull , Sara Kiesler, Connections: new ways of working in the networked organization, MIT Press, Cambridge, MA, 1991
	41	Stanney, K. M., and Maxey, J. (1997). Socially centered design. In G. Salvendy (Ed.), Handbook of Human Factors and Ergonomics (2nd Edition) (pp. 637--656). New York: Wiley.
	42	Lucy A. Suchman, Plans and situated actions: the problem of human-machine communication, Cambridge University Press, New York, NY, 1987
	43	Tashakkori, A., and Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: SAGE.
	44	Peter J. Thomas, CSCW Requirements and Evaluation, Springer-Verlag New York, Inc., Secaucus, NJ, 1996
	45	Michael Twidale , David Randall , Richard Bentley, Situated evaluation for cooperative systems, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.441-452, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193066]
	46	Vicente, K. J. (1997). Heeding the legacy of Meister, Brunswik, & Gibson: Toward a broader view of human factors research. Human Factors, 39(2), 323--328.
	47	Leon Watts , Andrew Monk , Owen Daly-Jones, Inter-personal awareness and synchronization: assessing the value of communication technologies, International Journal of Human-Computer Studies, v.44 n.6, p.849-873, June 1996 [doi>10.1006/ijhc.1996.0036]
	48	Steve Whittaker , Heinrich Schwarz, Meetings of the Board: The Impact of Scheduling Medium on Long Term GroupCoordination in Software Development, Computer Supported Cooperative Work, v.8 n.3, p.175-205, June 1999 [doi>10.1023/A:1008603001894]
	49	Woods, D. D. (2003). Discovering how distributed cognitive systems work. In E. Hollnagel (Ed.), Handbook of cognitive task design (pp. 37--53). Mahwah, N.J.: Lawarence Erlbaum.
	50	Xiao, Y., and Vicente, K. J. (2000). A framework for epistemological analysis in empirical (laboratory and field) studies. Human Factors, 42(1), 87--101.