A laboratory method for studying activity awareness

ABSTRACT

Many failures in long-term collaboration occur because of a lack of activity awareness. Activity awareness is a broad concept that involves awareness of synchronous and asynchronous interactions over extended time periods. We describe a procedure to evaluate activity awareness and collaborative activities in a controlled setting. The activities used are modeled on real-world collaborations documented earlier in a field study. We developed an experimental method to study these activity awareness problems in the laboratory. Participants worked on a simulated long-term project in the laboratory over multiple experimental sessions with a confederate, who partially scripted activities and probes. We present evidence showing that this method represents a valid model of real collaboration, based on participants' active engagement, lively negotiation, and awareness difficulties. We found that having the ability to define, reproduce, and systematically manipulate collaborative situations allowed us to assess the effect of realistic conditions on activity awareness in remote collaboration.

REFERENCES

• 1: Arrow H., McGrath, J. E. Berdahl, J. L., Small Groups as Complex Systems: Formation, Co-ordination, Development, and Adaptation. Sage Publications, 2000.
• 2: Baker, K. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. MSc Thesis, Department of Computer Science, University of Calgary, Alberta, Canada, 2002.
• 3: Campbell, D., & Fiske, D. W. Convergent and discriminate validation by the multitraitmultimethod matrix. Psychological Bulletin 4, (1959), 297--312.
• 4: John M. Carroll , Dennis C. Neale , Philip L. Isenhour , Mary Beth Rosson , D. Scott McCrickard, Notification and awareness: synchronizing task-oriented collaborative activity, International Journal of Human-Computer Studies, v.58 n.5, p.605-632, May 2003 [doi>10.1016/S1071-5819(03)00024-7]
• 5: Clark, H. H. Using Language. New York: Cambridge University Press, 1996.
• 6: Gilbert Cockton , Alan Woolrych, Sale must end: should discount methods be cleared off HCI's shelves?, interactions, v.9 n.5, September 2002 [doi>10.1145/566981.566990]
• 7: T. Connolly , L. M. Jessup , J. S. Valacich, Effects of anonymity and evaluative tone on idea generation in computer-mediated groups, Management Science, v.36 n.6, p.689-703, Jun. 1990 [doi>10.1287/mnsc.36.6.689]
• 8: Steve M. Easterbrook, Cooperation or Conflict?, Springer-Verlag New York, Inc., Secaucus, NJ, 1993
• 9: Endsley, M. R., Toward a theory of situation awareness in dynamic systems. Human Factors 37, (1995), 32--64.
• 10: Flanagan, J. C. 1954. The critical incident technique. Psychological bulletin 51(4), (1954), 327--358.
• 11: Craig H. Ganoe , Jacob P. Somervell , Dennis C. Neale , Philip L. Isenhour , John M. Carroll , Mary Beth Rosson , D. Scott McCrickard, Classroom BRIDGE: using collaborative public and desktop timelines to support activity awareness, Proceedings of the 16th annual ACM symposium on User interface software and technology, p.21-30, November 02-05, 2003, Vancouver, Canada [doi>10.1145/964696.964699]
• 12: Groove Networks Inc., http://www.Grove.com, 2003.
• 13: Jonathan Grudin, Why CSCW applications fail: problems in the design and evaluation of organization of organizational interfaces, Proceedings of the 1988 ACM conference on Computer-supported cooperative work, p.85-93, September 26-28, 1988, Portland, Oregon, United States [doi>10.1145/62266.62273]
• 14: Carl Gutwin , Saul Greenberg, The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces, Proceedings of the 9th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, p.98-103, June 04-16, 2000
• 15: Gutwin, C. & Greenberg, S. Workspace Awareness Position paper for the ACM CHI'97 workshop on Awareness in Collaborative Systems, organized by McDaniel S. E., & Brinck T., Atlanta, Georgia (1997).
• 16: Carl Gutwin , Saul Greenberg, Workspace awareness for groupware, Conference companion on Human factors in computing systems: common ground, p.208-209, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257284]
• 17: Hayes-Roth, B. & Hayes-Roth, F. A cognitive model of planning. Cognitive Science, 3, (1979), 275--310.
• 18: Kari Kuutti , Tuula Arvonen, Identifying potential CSCW applications by means of activity theory concepts: a case example, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.233-240, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.150955]
• 19: Joseph E. McGrath, Methodology matters: doing research in the behavioral and social sciences, Human-computer interaction: toward the year 2000, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 20: Mohammed, S., & Dumville, B., Team Mental Models: Expanding Theory and Measurement Through Cross-Disciplinary Boundaries. Journal of Organizational Behavior, 22 (2001), 89--106.
• 21: Dennis C. Neale , John M Carroll , Mary Beth Rosson, Evaluating computer-supported cooperative work: models and frameworks, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031626]
• 22: Neale, D. C., Carroll, J. M., 1999. Multi-faceted evaluation for complex, distributed activities. Proc. of the CSCL'99 Computer Supported Co-operative Learning. Lawrence Erlbaum, 425--433.
• 23: David Pinelle , Carl Gutwin, Groupware walkthrough: adding context to groupware usability evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503458]
• 24: David Pinelle , Carl Gutwin , Saul Greenberg, Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.4, p.281-311, December 2003 [doi>10.1145/966930.966932]
• 25: Wolfgang Prinz , Gloria Mark , Uta Pankoke-Babatz, Designing groupware for congruency in use, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.373-382, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289512]
• 26: Ramage M., The Learning Way: Evaluating Co-operatives Systems, PhD thesis, Lancaster Univ., 1999.
• 27: Salas, E., Prince, C., Baker, D. P., & Shrestha, L., Situation awareness in team performance: Implications for measurements and training, Human Factors, 37 (1995), 123--136.
• 28: Scriven, M. The methodology of evaluation. In R. Tyler, R. Gagne & M. Scriven (Eds.), Perspectives of curriculum evaluation. Chicago: Rand McNally, p. 39--83, 1967.
• 29: Michelle Potts Steves , Emile Morse , Carl Gutwin , Saul Greenberg, A comparison of usage evaluation and inspection methods for assessing groupware usability, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA [doi>10.1145/500286.500306]
• 30: Suchman, L. A. Plans and situated actions. New York: Cambridge University Press, 1986.
• 31: Michael Twidale , David Randall , Richard Bentley, Situated evaluation for cooperative systems, Proceedings of the 1994 ACM conference on Computer supported cooperative work, p.441-452, October 22-26, 1994, Chapel Hill, North Carolina, United States [doi>10.1145/192844.193066]
• 32: Leon Watts , Andrew Monk , Owen Daly-Jones, Inter-personal awareness and synchronization: assessing the value of communication technologies, International Journal of Human-Computer Studies, v.44 n.6, p.849-873, June 1996 [doi>10.1006/ijhc.1996.0036]
• 33: Weisband, S., Overcoming social awareness in computer-supported groups: Does anonymity really help?, Proc. CSCW'94 (2) (1994), 285--297.
• 34: Terry Winograd , Fernando Flores, Understanding Computers and Cognition: A New Foundation for Design, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1987

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

Keywords:
CSCW, activity awareness, awareness, coordination