Empirical evidence of the benefits of workspace awareness in software configuration management

ABSTRACT

In this paper, we present results from our empirical evaluations of a workspace awareness tool that we designed and implemented to augment the functionality of software configuration management systems. Particularly, we performed two user experiments directed at understanding the effectiveness of a workspace awareness tool in improving coordination and reducing conflicts. In the first experiment, we evaluated the tool through text-based assignments to avoid interference from the well-documented impact of individual differences among participants, as these differences are known to lessen the observable effect of proposed tools or to lead to them having no observable effect at all. This strategy of evaluating an application in a domain that is known to have less individual differences is novel and in our case particularly helpful in providing baseline quantifiable results. Upon this baseline, we performed a second experiment, with code-based assignments, to validate that the tool's beneficial effects also occur in the case of programming. Together, our results provide quantitative evidence of the benefits of workspace awareness in software configuration management, as we demonstrate that it improves coordination and conflict resolution without inducing significant overhead in monitoring awareness cues.

REFERENCES

• 1: Paul Dourish , Victoria Bellotti, Awareness and coordination in shared workspaces, Proceedings of the 1992 ACM conference on Computer-supported cooperative work, p.107-114, November 01-04, 1992, Toronto, Ontario, Canada [doi>10.1145/143457.143468]
• 2: Christian Heath , Marina Jirotka , Paul Luff , Jon Hindmarsh, Unpacking collaboration: the interactional organisation of trading in a city dealing room, Computer Supported Cooperative Work, v.3 n.2, p.147-165, 1994/1995 [doi>10.1007/BF00773445]
• 3: Rebecca E. Grinter , James D. Herbsleb , Dewayne E. Perry, The geography of coordination: dealing with distance in R&D work, Proceedings of the international ACM SIGGROUP conference on Supporting group work, p.306-315, November 14-17, 1999, Phoenix, Arizona, United States [doi>10.1145/320297.320333]
• 4: Carl Gutwin , Reagan Penner , Kevin Schneider, Group awareness in distributed software development, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031621]
• 5: Carl Gutwin , Saul Greenberg, Workspace awareness for groupware, Conference companion on Human factors in computing systems: common ground, p.208-209, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257284]
• 6: Marcelo Cataldo , Patrick A. Wagstrom , James D. Herbsleb , Kathleen M. Carley, Identification of coordination requirements: implications for the Design of collaboration and awareness tools, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180929]
• 7: James D. Herbsleb , Rebecca E. Grinter, Architectures, Coordination, and Distance: Conway's Law and Beyond, IEEE Software, v.16 n.5, p.63-70, September 1999 [doi>10.1109/52.795103]
• 8: Rebecca E. Grinter, Supporting articulation work using software configuration management systems, Computer Supported Cooperative Work, v.5 n.4, p.447-465, 1996 [doi>10.1007/BF00136714]
• 9: Dewayne E. Perry , Harvey P. Siy , Lawrence G. Votta, Parallel changes in large-scale software development: an observational case study, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.3, p.308-337, July 2001 [doi>10.1145/383876.383878]
• 10: Cleidson R. B. de Souza , David Redmiles , Li-Te Cheng , David Millen , John Patterson, Sometimes you need to see through walls: a field study of application programming interfaces, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031620]
• 11: Davor Cubranic , Gail C. Murphy , Janice Singer , Kellogg S. Booth, Hipikat: A Project Memory for Software Development, IEEE Transactions on Software Engineering, v.31 n.6, p.446-465, June 2005 [doi>10.1109/TSE.2005.71]
• 12: Jacky Estublier , Sergio Garcia, Process model and awareness in SCM, Proceedings of the 12th international workshop on Software configuration management, p.59-74, September 05-06, 2005, Lisbon, Portugal [doi>10.1145/1109128.1109133]
• 13: Ciaran O'Reilly , David Bustard , Philip Morrow, The war room command console: shared visualizations for inclusive team coordination, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056026]
• 14: Jacob T. Biehl , Mary Czerwinski , Greg Smith , George G. Robertson, FASTDash: a visual dashboard for fostering awareness in software teams, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240823]
• 15: Dewan, P. and R. Hegde. 2007. Semi-Synchronous Conflict Detection and Resolution in Asynchronous Software Development. European Computer Supported Cooperative Work. p. 159--178.
• 16: Anita Sarma , Gerald Bortis , Andre van der Hoek, Towards supporting awareness of indirect conflicts across software configuration management workspaces, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA [doi>10.1145/1321631.1321647]
• 17: Mayer, R. E., 1988. From Novice to Expert, in Handbook of Human-Computer Interaction, M. G. Helander, T. K. Landauer, and P. Prabhu, Editors. p. 781--795.
• 18: David F. Redmiles, Reducing the variability of programmers' performance through explained examples, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.67-73, May 1993, Amsterdam, The Netherlands
• 19: Anita Sarma , Andre van der Hoek , David F. Redmiles, A Comprehensive Evaluation of Workspace Awareness in Software Configuration Management Systems, Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, p.23-26, September 23-27, 2007 [doi>10.1109/VLHCC.2007.1]
• 20: Rebecca E. Grinter, Recomposition: putting it all back together again, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.393-402, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289514]
• 21: T. Mens, A State-of-the-Art Survey on Software Merging, IEEE Transactions on Software Engineering, v.28 n.5, p.449-462, May 2002 [doi>10.1109/TSE.2002.1000449]
• 22: Anita Sarma , Zahra Noroozi , André van der Hoek, Palantír: raising awareness among configuration management workspaces, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 23: Cleidson R. B. de Souza , David Redmiles , Paul Dourish, "Breaking the code", moving between private and public work in collaborative software development, Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, November 09-12, 2003, Sanibel Island, Florida, USA [doi>10.1145/958160.958177]
• 24: Gary M. Olson , Judith S. Olson, Distance matters, Human-Computer Interaction, v.15 n.2, p.139-178, September 2000 [doi>10.1207/S15327051HCI1523_4]
• 25: Rebecca E. Grinter, Using a configuration management tool to coordinate software development, Proceedings of conference on Organizational computing systems, p.168-177, August 13-16, 1995, Milpitas, California, United States [doi>10.1145/224019.224036]
• 26: Berliner, B. 1990. CVS II: Parallelizing Software Development. USENIX Winter 1990 Tech. Conference. p. 341--352.
• 27: Li-Te Cheng , Susanne Hupfer , Steven Ross , John Patterson, Jazzing up Eclipse with collaborative tools, Proceedings of the 2003 OOPSLA workshop on eclipse technology eXchange, p.45-49, October 27-27, 2003, Anaheim, California [doi>10.1145/965660.965670]
• 28: Margaret-Anne D. Storey , Davor Čubranić , Daniel M. German, On the use of visualization to support awareness of human activities in software development: a survey and a framework, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056045]
• 29: Wolfgang Appelt, WWW Based Collaboration with the BSCW System, Proceedings of the 26th Conference on Current Trends in Theory and Practice of Informatics on Theory and Practice of Informatics, p.66-78, November 27-December 04, 1999
• 30: Pascal Molli , Hala Skaf-Molli , Christophe Bouthier, State Treemap: An Awareness Widget for Multi-Synchronous Groupware, Proceedings of the Seventh International Workshop on Groupware, p.106-114, September 06-08, 2001
• 31: Till Schümmer , Jörg M. Haake, Supporting distributed software development by modes of collaboration, Proceedings of the seventh conference on European Conference on Computer Supported Cooperative Work, p.79-98, September 16-20, 2001, Bonn, Germany
• 32: Brian de Alwis , Gail C. Murphy , Martin P. Robillard, A Comparative Study of Three Program Exploration Tools, Proceedings of the 15th IEEE International Conference on Program Comprehension, p.103-112, June 26-29, 2007 [doi>10.1109/ICPC.2007.6]
• 33: Shadish, W. R., T. D. Cook, and D. T. Campbell, 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 1 ed: Houghton Mifflin Company. pp. 623.
• 34: Brooks Jr., F. P, 1974. The Mythical Man-Month. Datamation, 20(12): p. 44--52.
• 35: James D. Herbsleb , Audris Mockus , Thomas A. Finholt , Rebecca E. Grinter, Distance, dependencies, and delay in a global collaboration, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.319-328, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.359003]
• 36: Bill Curtis , Herb Krasner , Neil Iscoe, A field study of the software design process for large systems, Communications of the ACM, v.31 n.11, p.1268-1287, Nov. 1988 [doi>10.1145/50087.50089]
• 37: Cleidson R. B. de Souza , David Redmiles , Li-Te Cheng , David Millen , John Patterson, How a good software practice thwarts collaboration: the multiple roles of APIs in software development, Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, October 31-November 06, 2004, Newport Beach, CA, USA

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Programmer workbench

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Version control
      D.2.9 Management
          Subjects: Software configuration management

General Terms:
Experimentation, Human Factors, Management

Keywords:
conflicts, evaluation, parallel work, software configuration management, user experiments, workspace awareness