Situating evaluation in scenarios of use

ABSTRACT

We report on the use of scenario-based methods for evaluating collaborative systems. We describe the method, the case study where it was applied, and provide results of its efficacy in the field. The results suggest that scenario-based evaluation is effective in helping to focus evaluation efforts and in identifying the range of technical, human, organizational and other contextual factors that impact system success. The method also helps identify specific actions, for example, prescriptions for design to enhance system effectiveness. However, we found the method somewhat less useful for identifying the measurable benefits gained from a CSCW implementation, which was one of our primary goals. We discuss challenges faced applying the technique, suggest recommendations for future research, and point to implications for practice.

REFERENCES

• 1: Ackerman, M., (2000) The Intellectual Challenge of CSCW: The gap between social requirements and technical feasibility. Human-Computer Interaction, 15, 179--203.
• 2: Kevin Baker , Saul Greenberg , Carl Gutwin, Empirical development of a heuristic evaluation methodology for shared workspace groupware, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587093]
• 3: Bannon, L. (1994). Use, design and evaluation: steps towards an integration, in D. Shapiro et al. (eds), The Design of Computer Supported Cooperative Work and Groupware Systems, Amsterdam: Elsevier Science.
• 4: John M. Carroll, Making Use: Scenario-Based Design of Human-Computer Interactions, MIT Press, Cambridge, MA, 2000
• 5: John M. Carroll, Scenario-based design: envisioning work and technology in system development, John Wiley & Sons, Inc., New York, NY, 1995
• 6: John M. Carroll , Dennis C. Neale , Philip L. Isenhour , Mary Beth Rosson , D. Scott McCrickard, Notification and awareness: synchronizing task-oriented collaborative activity, International Journal of Human-Computer Studies, v.58 n.5, p.605-632, May 2003 [doi>10.1016/S1071-5819(03)00024-7]
• 7: John M. Carroll , Mary Beth Rosson, Getting around the task-artifact cycle: how to make claims and design by scenario, ACM Transactions on Information Systems (TOIS), v.10 n.2, p.181-212, April 1992 [doi>10.1145/146802.146834]
• 8: Gilbert Cockton , Alan Woolrych, Sale must end: should discount methods be cleared off HCI's shelves?, interactions, v.9 n.5, September 2002 [doi>10.1145/566981.566990]
• 9: Laurie Damianos , Lynette Hirschman , Robyn Kozierok , Jeffrey Kurtz , Andrew Greenberg , Kimberley Walls , Sharon Laskowski , Jean Scholtz, Evaluation for collaborative systems, ACM Computing Surveys (CSUR), v.31 n.2es, June 1999 [doi>10.1145/323216.323362]
• 10: Paul Dourish, Process descriptions as organisational accounting devices: the dual use of workflow technologies, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500297]
• 11: Wolfgang Dzida , Regine Freitag, Making Use of Scenarios for Validating Analysis and Design, IEEE Transactions on Software Engineering, v.24 n.12, p.1182-1196, December 1998 [doi>10.1109/32.738346]
• 12: Geertz, C. (1973). Description: Toward an Interpretive Theory of Culture. In The Interpretation of Culture, NY: Basic Books.
• 13: Jonathan Grudin, Why CSCW applications fail: problems in the design and evaluation of organization of organizational interfaces, Proceedings of the 1988 ACM conference on Computer-supported cooperative work, p.85-93, September 26-28, 1988, Portland, Oregon, United States [doi>10.1145/62266.62273]
• 14: Carl Gutwin , Mark Roseman , Saul Greenberg, A usability study of awareness widgets in a shared workspace groupware system, Proceedings of the 1996 ACM conference on Computer supported cooperative work, p.258-267, November 16-20, 1996, Boston, Massachusetts, United States [doi>10.1145/240080.240298]
• 15: Richard H. R. Harper, Looking at ourselves: an examination of the social organisation of two research laboratories, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.330-337, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143542]
• 16: John Hughes , Val King , Tom Rodden , Hans Andersen, Moving out from the control room: ethnography in system design, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.429-439, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193065]
• 17: Bonnie E. John, Evaluating usability evaluation techniques, ACM Computing Surveys (CSUR), v.28 n.4es, Dec. 1996 [doi>10.1145/242224.242402]
• 18: Rick Kazman , Gregory Abowd , Len Bass , Paul Clements, Scenario-Based Analysis of Software Architecture, IEEE Software, v.13 n.6, p.47-55, November 1996 [doi>10.1109/52.542294]
• 19: Kolbe, R. H. & Burnett, M. S. (1991). Content-analysis research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research, 18, 243--250.
• 20: Thomas K. Landauer, Trouble with Computers: Usefulness, Usability, and Productivity, MIT Press, Cambridge, MA, 1996
• 21: Lombard, M., Snyder-Dutch, J. and Bracken, C.C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4), 587--604.
• 22: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 23: Jakob Nielsen , Robert L. Mack, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 24: Olson, G and Olson, J. (1991). User-centered design of collaboration technology. Journal of Organizational Computing, 1(1), 61--83.
• 25: Wanda J. Orlikowski , Debra C. Gash, Technological frames: making sense of information technology in organizations, ACM Transactions on Information Systems (TOIS), v.12 n.2, p.174-207, April 1994 [doi>10.1145/196734.196745]
• 26: Leysia Palen, Social, individual and technological issues for groupware calendar systems, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.17-24, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.302982]
• 27: Pinelle, D. (2000) A Survey of Groupware Evaluations in CSCW Proceedings. Technical Report HCI-TR-2000-01, Computer Science Department, University of Saskatchewan.
• 28: David Pinelle , Carl Gutwin, Groupware walkthrough: adding context to groupware usability evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503458]
• 29: Ramage, M. (1999). The Learning Way: Evaluating Cooperative Systems. Unpublished PhD thesis, University of Lancaster. Available at http://www.comp.lancs.ac.uk/computing/research/cseg/projects/evaluation/index.html. Last accessed August 30, 2004.
• 30: Yvonne Rogers, Exploring obstacles: integrating CSCW in evolving organisations, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.67-77, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.192875]
• 31: Yvonne Rogers , Victoria Bellotti, Grounding blue-sky research: how can ethnography help?, interactions, v.4 n.3, p.58-63, May/June 1997 [doi>10.1145/255392.255404]
• 32: Ross, S., Ramage, M., and Rogers, Y. (1995). PETRA: Participatory Evaluation Through Redesign and Analysis. Interacting with Computers, 7(4): 335--360.
• 33: Mary Beth Rosson , John M. Carroll, Usability engineering: scenario-based development of human-computer interaction, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001
• 34: Peter B. Seddon , Valerie Graeser , Leslie P. Willcocks, Measuring organizational IS effectiveness: an overview and update of senior management perspectives, ACM SIGMIS Database, v.33 n.2, Spring 2002 [doi>10.1145/513264.513270]
• 35: Oliver Stiemerling , Armin B. Cremers, The Use of Cooperation Scenarios in the Design and Evaluation of a CSCW System, IEEE Transactions on Software Engineering, v.24 n.12, p.1171-1181, December 1998 [doi>10.1109/32.738345]
• 36: Michelle Potts Steves , Emile Morse , Carl Gutwin , Saul Greenberg, A comparison of usage evaluation and inspection methods for assessing groupware usability, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500306]
• 37: Lucy A. Suchman, Plans and situated actions: the problem of human-machine communication, Cambridge University Press, New York, NY, 1987
• 38: Michael Twidale , David Randall , Richard Bentley, Situated evaluation for cooperative systems, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.441-452, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193066]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Evaluation/methodology; Computer-supported cooperative work

General Terms:
Human Factors, Performance

Keywords:
CSCW, computer-supported cooperative work, evaluation, scenario-based evaluation