Situated evaluation for cooperative systems

ABSTRACT

This paper discusses an evaluation of the MEAD prototype, a multi-user interface generator tool particularly for use in the context of Air Traffic Control (ATC). The procedures we adopted took the form of opportunistic and informal evaluation sessions with small user groups, including Air Traffic Controllers (ATCOs). We argue that informal procedures are a powerful and cost effective method for dealing with specific evaluation issues in the context of CSCW but that wider issues are more problematic. Most notably, identifying the “validity” or otherwise of CSCW systems requires that the context of use be taken seriously, necessitating a fundamental re-appraisal of the concept of evaluation.

REFERENCES

• 1: Bannon, L. Use, design and evaluation: Steps towards an integration. In Proceedings of the International Workshop on the Design of CSCW and Groupware Systems. To appear in D. Shapiro, et al. (Eds.), The Design of Computer Supported Cooperative Work and Groupware Systems. Elsevier Science, Amsterdam, forthcoming.
• 2: Barnard, P. Applied cognitive psychology: Research for human-computer interaction. In A. Downton (FA.), Engineering the Human-Computer Interface (pp. 28- 61). McGraw-Hill, Maidenhead, 1991.
• 3: Bentley, R. Supporting multi-user interface development for cooperative systems. Unpublished doctoral dissertation, available as a technical report. Computing Department, Lancaster University, January 1994.
• 4: R. Bentley , J. A. Hughes , D. Randall , T. Rodden , P. Sawyer , D. Shapiro , I. Sommerville, Ethnographically-informed systems design for air traffic control, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.123-129, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143470]
• 5: Richard Bentley , Tom Rodden , Peter Sawyer , Ian Sommerville, An architecture for tailoring cooperative multi-user displays, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.187-194, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143478]
• 6: Susanne Bø , Kaj Grøbæk, Cooperative prototyping: users and designers in mutual activity, International Journal of Man-Machine Studies, v.34 n.3, p.453-478, March 1991 [doi>10.1016/0020-7373(91)90030-B]
• 7: C. Cool , R. S. Fish , R. E. Kraut , C. M. Lowery, Iterative design of video communication systems, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.25-32, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143458]
• 8: Davenport, T. Process Innovation. Ernst and Young, London, 1993.
• 9: Downton, A. Evaluation techniques for humancomputer systems design. In A. Downton (Ed.), Engineering the Human-Computer Interface (pp. 325- 355). McGraw-Hill, Maidenhead, 1991.
• 10: Feigenbaum, A. V. Total Quality Control (3rd ed.). McGraw-Hill, New York, 1991.
• 11: Draper, S., and Wately, D. Practical methods for measuring the performance of human-computer interfaces. Distributed notes accompanying talk at the JCI HCI Summer School, Queen Mary College, Aug. 1991.
• 12: Jonathan Grudin, Why CSCW applications fail: problems in the design and evaluation of organization of organizational interfaces, Proceedings of the 1988 ACM conference on Computer-supported cooperative work, p.85-93, September 26-28, 1988, Portland, Oregon, United States [doi>10.1145/62266.62273]
• 13: Hammersley, M., and Atkinson, P. Ethnography Principles in Practice. Routledge, London, 1983.
• 14: R. R. Harper , J. A. Hughes , D. Z. Shapiro, Harmonious working and CSCW: computer technology and air traffic control, Studies in computer supported cooperative work: theory, practice and design, North-Holland Publishing Co., Amsterdam, The Netherlands, 1990
• 15: Heath, C., and Luff, P. Collaborative activity and technological design: Task coordination in London underground control rooms. In Proceedings of ECSCW'91 (Amsterdam), 1991, pp. 65-80.
• 16: Hughes, J. A., Shapiro, D., and Randall, D. CSCW: Discipline or paradigm? In Proceedings of ECSCW'91 (Amsterdam), 1991, pp. 309-323.
• 17: John A. Hughes , David Randall , Dan Shapiro, Faltering from ethnography to design, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.115-122, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143469]
• 18: Solange Karsenty , James A. Landay , Chris Weikart, Inferring graphical constraints with Rockit, Proceedings of the conference on People and computers VII, p.137-153, January 1993, York, United Kingdom
• 19: Thomas K. Landauer, Relations between cognitive psychology and computer system design, Interfacing thought: cognitive aspects of human-computer interaction, MIT Press, Cambridge, MA, 1987
• 20: Jakob Nielsen, Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.), p.394-401, September 1989, Boston, Massachusetts, United States
• 21: Randall, D. Perspectives on evaluation: Putting your money where your mouth is. Paper presented at the UK CSCW SIG (Sept., DTI, London), 1993. (Available from the author.)
• 22: Randall, D., Hughes, J. A., and Shapiro, D. Systems development--the fourth dimension: Perspectives on the social organisation of work. In P. Quintas (Ed.), The Social Dimensions of Software Engineering. Ellis Horwood, London and New York, 1993.
• 23: RSRE, Prototype Electronic Flight Strips {Video}. Malvern, 1991.
• 24: Self, J. Journal of Artificial intelligence and Education 4, 2-3, 1993. (Special issue on evaluation.)
• 25: Shapiro, D., Hughes, J. A., Randall, D., and Harper, R. Visual re-representation of database information: The flight strip in air traffic control. In M. J. Tauber, et. al. (Eds.), Cognitive Aspects of Visual Languages and Visual Interfaces (pp. 349-376). Elsevier Science, Amsterdam, 1994.
• 26: Suchman, L. Plans and Situated Actions. Cambridge University Press, Cambridge, 1987.
• 27: Deborah G. Tatar , Gregg Foster , Daniel G. Bobrow, Design for conversation: lessons from Cognoter, International Journal of Man-Machine Studies, v.34 n.2, p.185-209, Feb. 1991 [doi>10.1016/0020-7373(91)90041-5]
• 28: Twidale, M. B. Redressing the balance: The advantages of informal evaluation techniques for intelligent learning environments. In J. of A1 and Ed. 4 , 2-3, 1993, 155-178.
• 29: Twidale, M. B., Rodden, T., and Sommerville, I. The Designers' Notepad: Supporting and understanding cooperative design. In Proceedings of ECSCW'93 (Milan, Italy), 1993, pp. 93-108.
• 30: Twidale, M. B., Rodden, T., and Sommerville, I. Developing a tool to support collaborative dialogues and graphical representation of ideas. In F. Verdejo (Ed.), Collaborative Dialogue Technologies in Distance Learning, in press.
• 31: Vincent, D. The Information Based Corporation. Dow Jones Irwin, 1990.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

General Terms:
Theory

Keywords:
air traffic control, ethnographic observation, multiuser interface design, rapid prototyping