"...real, concrete facts about what works...": integrating evaluation and design through patterns

ABSTRACT

Recent CSCW research has focused on methods for evaluating usability, rather than the more problematic evaluation of systems in use. A possible approach to the integration of use, design and evaluation is through the representation of evaluation findings as design-oriented models. A method is described for modeling computer-supported cooperative work and its context: a design patterns language, based on the principles of activity theory. The language is the outcome of an evaluation of the evolving use of tools to support collaborative information sharing, carried out at a global NGO.

REFERENCES

• 1: Alexander, C., Silverstein, M., Angel, S., Ishikawa, S. and Abrams, D. The Oregon Experiment. Oxford University Press, New York, 1975.
• 2: Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I. and Angel, S. A Pattern Language: Towns, Buildings, Construction. Oxford University Press, New York, 1977.
• 3: Alexander, C. The Timeless Way of Building. Oxford University Press, New York, 1979.
• 4: Kevin Baker , Saul Greenberg , Carl Gutwin, Empirical development of a heuristic evaluation methodology for shared workspace groupware, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587093]
• 5: Bannon, L. Use, Design and Evaluation: Steps Towards an Integration. In: Shapiro, D., M. Tauber and R. Traunmuller, Eds., The Design of Computer Supported Cooperative Work and Groupware Systems. Elsevier, Amsterdam, 1996. 4230--443.
• 6: Jørgen P. Bansler , Erling Havn, Technology-use mediation: making sense of electronic communication in an organizational context, Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, November 09-12, 2003, Sanibel Island, Florida, USA [doi>10.1145/958160.958182]
• 7: Bertelsen, O. and Bodker, S. Activity Theory. In: Carroll, J. M., Ed. HCI Models, Theories and Frameworks: Towards an Interdisciplinary Science. Morgan Kaufmann Publishers, San Francisco, CA, 2003. 291--314.
• 8: Susanne Bødker, Applying activity theory to video analysis: how to make sense of video data in human-computer interaction, Context and consciousness: activity theory and human-computer interaction, Massachusetts Institute of Technology, Cambridge, MA, 1995
• 9: Bodker, S. and Christiansen, E. Scenarios as Springboards in CSCW Design. In: Bowker, G., S. L. Star, W. Turner and L. Gasser, Eds., Social Science, Technical Systems and Cooperative Work: Beyond the Great Divide. Lawrence Erlbaum Associates, Mahwah, NJ, 1997. 217--233.
• 10: Andy Crabtree , Terry Hemmings , Tom Rodden, Pattern-based support for interactive design in domestic settings, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, June 25-28, 2002, London, England [doi>10.1145/778712.778749]
• 11: Dearden, A., Finlay, J., Allgar, E. and McManus, B. Using Pattern Languages in Participatory Design. In Proceedings of the Participatory Design Conference. (PDC 2002, Malmo, Sweden, June 23-25, 2002.) 104--113.
• 12: Dearden, A. and Finlay, J. Pattern Languages in HCI: A Critical Review. Human-Computer Interaction (accepted).
• 13: Engestrom, Y. Learning by Expanding: An Activity-Theoretical Approach to Developmental Work Research. Orienta-Konsultit Oy, Helsinki, 1987.
• 14: Erickson, T. Supporting Interdisciplinary Design: Towards Pattern Languages for Workplaces. In: Luff, P., J. Hindmarsh and C. Heath, Eds., Workplace Studies: Recovering Work Practice and Informing System Design. Cambridge University Press, Cambridge, 2000. 252--261.
• 15: Thomas Erickson, Lingua Francas for design: sacred places and pattern languages, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.357-368, August 17-19, 2000, New York City, New York, United States [doi>10.1145/347642.347794]
• 16: Greenberg, S., Fitzpatrick, G., Gutwin, C. and Kaplan, S. Adapting the Locales Framework for Heuristic Evaluation of Groupware. In Proceedings of the Australian Conference on Human Computer Interaction. (OZCHI'99, Wagga Wagga, NSW, November 28-30,1999.) 30--36.
• 17: Jonathan Grudin, Why CSCW applications fail: problems in the design and evaluation of organization of organizational interfaces, Proceedings of the 1988 ACM conference on Computer-supported cooperative work, p.85-93, September 26-28, 1988, Portland, Oregon, United States [doi>10.1145/62266.62273]
• 18: Guy, E. S. Appropriating Patterns for the Activity Theory Toolkit. In Proceedings of the First International Workshop on Activity Theory Based Practical Methods for IT Design. (ATIT 2004, in association with the Third Nordic Conference on Cultural and Activity Research, Copenhagen, Denmark, September 2-5, 2004.) 33--48.
• 19: Guy, E. S. Patterns of Activity: Modelling Computer Supported Cooperative Work. PhD Thesis, University of Brighton, UK, 2005.
• 20: Steven R. Haynes , Sandeep Purao , Amie L. Skattebo, Situating evaluation in scenarios of use, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031624]
• 21: Thomas Herrmann , Marcel Hoffmann , Isa Jahnke , Andrea Kienle , Gabriele Kunau , Kai-Uwe Loser , Natalja Menold, Concepts for usable patterns of groupware applications, Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, November 09-12, 2003, Sanibel Island, Florida, USA [doi>10.1145/958160.958216]
• 22: Hudson, M. Managing Without Profit: The Art of Managing Third-Sector Organizations. Penguin Books, London, 1995.
• 23: Hughes, J., O'Brien, J., Rouncefield, M. and Rodden, T. "They're Supposed to Be Fixing it": Requirements and System Redesign. In: Thomas, P., Ed. CSCW Requirements and Evaluation. Springer-Verlag, London, 1996. 21--38.
• 24: Leont'ev, A. N. The Problem of Activity in Psychology. In: Wertsch, J. V., Ed. The Concept of Activity in Soviet Psychology. M. E. Sharp Inc., New York, 1979. 37--71.
• 25: Martin, D., Rodden, T., Rouncefield, M., Sommerville, I. and Viller, S. Finding Patterns in the Fieldwork. In Proceedings of the Seventh European Conference on Computer-Supported Cooperative Work. (ECSCW 2001, Bonn, 16-20 September, 2001.) 39--57.
• 26: Dennis C. Neale , John M Carroll , Mary Beth Rosson, Evaluating computer-supported cooperative work: models and frameworks, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031626]
• 27: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 28: Kazuo Okamura , Wanda J. Orlikowski , Masayo Fujimoto , JoAnne Yates, Helping CSCW applications succeed: the role of mediators in the context of use, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.55-65, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192871]
• 29: Wanda J. Orlikowski, Learning from Notes: organizational issues in groupware implementation, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.362-369, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143549]
• 30: David Pinelle , Carl Gutwin, Groupware walkthrough: adding context to groupware usability evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503458]
• 31: Michelle Potts Steves , Emile Morse , Carl Gutwin , Saul Greenberg, A comparison of usage evaluation and inspection methods for assessing groupware usability, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500306]
• 32: Ross, S., Ramage, M. and Rogers, Y. PETRA: Participatory Evaluation Through Redesign and Analysis. Interacting with Computers 7, 4, (1995), 335--360.
• 33: Schmidt, K. Riding a Tiger, or Computer Supported Cooperative Work. In Proceedings of the Second European Conference on Computer-Supported Cooperative Work. (ECSCW '91, Amsterdam, September 25-27, 1991.) 1--16.
• 34: Scriven, M. The Methodology of Evaluation. In: Tyler, R., R. Gagne and M. Scriven, Eds., Perspectives of Curriculum Evaluation. Rand McNally & Co, Chicago, 1967. 39--83.
• 35: Oliver Stiemerling , Armin B. Cremers, The Use of Cooperation Scenarios in the Design and Evaluation of a CSCW System, IEEE Transactions on Software Engineering, v.24 n.12, p.1171-1181, December 1998 [doi>10.1109/32.738345]
• 36: Michael Twidale , David Randall , Richard Bentley, Situated evaluation for cooperative systems, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.441-452, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193066]
• 37: Vygotsky, L. S., Thought and Language. Cambridge, MA: MIT Press, 1962.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Groupware

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Evaluation/methodology; Computer-supported cooperative work; Organizational design

General Terms:
Design, Documentation, Theory

Keywords:
activity patterns, activity theory, evaluation, modeling computer-supported cooperative work, pattern languages