Latent social structure in open source projects

ABSTRACT

Commercial software project managers design project organizational structure carefully, mindful of available skills, division of labour, geographical boundaries, etc. These organizational "cathedrals" are to be contrasted with the "bazaar-like" nature of Open Source Software (OSS) Projects, which have no pre-designed organizational structure. Any structure that exists is dynamic, self-organizing, latent, and usually not explicitly stated. Still, in large, complex, successful, OSS projects, we do expect that subcommunities will form spontaneously within the developer teams. Studying these subcommunities, and their behavior can shed light on how successful OSS projects self-organize. This phenomenon could well hold important lessons for how commercial software teams might be organized. Building on known well-established techniques for detecting community structure in complex networks, we extract and study latent subcommunities from the email social network of several projects: Apache HTTPD, Python, PostgresSQL, Perl, and Apache ANT. We then validate them with software development activity history. Our results show that subcommunities do indeed spontaneously arise within these projects as the projects evolve. These subcommunities manifest most strongly in technical discussions, and are significantly connected with collaboration behaviour.

REFERENCES

• 1: Manju K. Ahuja , Dennis F. Galletta , Kathleen M. Carley, Individual Centrality and Performance in Virtual R&D Groups: An Empirical Study, Management Science, v.49 n.1, p.21-38, January 2003 [doi>10.1287/mnsc.49.1.21.12756]
• 2: T. Allen et al. Managing the flow of technology. Cambridge: The MIT Pr., 1979.
• 3: U. Alon. Biological Networks: The Tinkerer as an Engineer. Science, 301(5641):1866--1867, 2003.
• 4: Lars Backstrom , Dan Huttenlocher , Jon Kleinberg , Xiangyang Lan, Group formation in large social networks: membership, growth, and evolution, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, August 20-23, 2006, Philadelphia, PA, USA [doi>10.1145/1150402.1150412]
• 5: Y. Benjamini and Y. Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289--300, 1995.
• 6: J. Berkus. The 5 types of open source projects. March 20, 2007 http://www.powerpostgresql.com/5_types.
• 7: Christian Bird , Alex Gourley , Prem Devanbu , Michael Gertz , Anand Swaminathan, Mining email social networks, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138016]
• 8: Christian Bird , Alex Gourley , Prem Devanbu , Michael Gertz , Anand Swaminathan, Mining email social networks, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138016]
• 9: Christian Bird , Alex Gourley , Prem Devanbu , Anand Swaminathan , Greta Hsu, Open Borders? Immigration in Open Source Projects, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.6, May 20-26, 2007 [doi>10.1109/MSR.2007.23]
• 10: G. Box, W. Hunter, and J. Hunter. Statistics for experimenters: an introductory to design data analysis and model building. Wiley Series in Probability and Mathematical Statistics)., 1978.
• 11: P. Boykin and V. Roychowdhury. Personal Email Networks: An Effective Anti-Spam Tool. Arxiv preprint cond-mat/0402143, 2004.
• 12: Frederick P. Brooks, Jr., The mythical man-month (anniversary ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 13: Marcelo Cataldo , Patrick A. Wagstrom , James D. Herbsleb , Kathleen M. Carley, Identification of coordination requirements: implications for the Design of collaboration and awareness tools, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180929]
• 14: A. Clauset, M. E. J. Newman, and C. Moore. Finding community structure in very large networks. Physical Review E, 70(6):66111, 2004.
• 15: J. F. P. D. Cleidson de Souza. Seeking the source: Software source code as a social and technical artifact, 2005. http://opensource.mit.edu/papers/desouza.pdf.
• 16: M. Conway. How do committees invent. Datamation, 14(4):28--31, 1968.
• 17: K. Crowston and J. Howison. The social structure of free and open source software development. First Monday, 10(2), 2005.
• 18: Bill Curtis , Herb Krasner , Neil Iscoe, A field study of the software design process for large systems, Communications of the ACM, v.31 n.11, p.1268-1287, Nov. 1988 [doi>10.1145/50087.50089]
• 19: P. Dalgaard. Introductory Statistics With R. Springer, 2002.
• 20: L. Danon, A. Diaz-Guilera, J. Duch, and A. Arenas. Comparing community structure identification. Journal of Statistical Mechanics: Theory and Experiment, 9:P09008, 2005.
• 21: S. N. Dorogovtsev , J. F. F. Mendes, Evolution of Networks: From Biological Nets to the Internet and WWW (Physics), Oxford University Press, Inc., New York, NY, 2003
• 22: Nicolas Ducheneaut, Socialization in an Open Source Software Community: A Socio-Technical Analysis, Computer Supported Cooperative Work, v.14 n.4, p.323-368, August 2005 [doi>10.1007/s10606-005-9000-1]
• 23: Nicolas Ducheneaut , Leon A. Watts, In search of coherence: a review of e-mail research, Human-Computer Interaction, v.20 n.1, p.11-48, June 2005 [doi>10.1207/s15327051hci2001&2_2]
• 24: Kate Ehrlich , Klarissa Chang, Leveraging expertise in global software teams: Going outside boundaries, Proceedings of the IEEE international conference on Global Software Engineering, p.149-158, October 16-19, 2006
• 25: Michael R. Garey , David S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, W. H. Freeman & Co., New York, NY, 1979
• 26: M. Girvan and M. E. J. Newman. Community structure in social and biological networks. PROC. NATL. ACAD. SCI. USA, 99:7821, 2002.
• 27: C. Gkantsidis, M. Mihail, and E. Zegura. The markov chain simulation method for generating connected power law random graphs. In Proceedings of ALENEX '03, pages 16--25, 2003.
• 28: P. Gleiser and L. Danon. Community structure in jazz. Advances in Complex Systems, 6:565, 2003.
• 29: J. González-Barahona, L. López, and G. Robles. Community structure of modules in the apache project. In MSR '05: Proceedings of the 2005 international workshop on Mining software repositories, 2005.
• 30: R. Guimera, L. Danon, A. Diaz-Guilera, F. Giralt, and A. Arenas. Self-similar community structure in organisations. Physical Review E, 68:065103, 2003.
• 31: R. Guimerà, S. Mossa, A. Turtschi, and L. Amaral. From the Cover: The worldwide air transportation network: Anomalous centrality, community structure, and cities' global roles. Proc Natl Acad Sci US A, 102(22):7794--7799, 2005.
• 32: R. M. Henderson and K. B. Clark. Architectural innovation: The reconfiguration of existing product technologies and the failure of established firms. Administrative Science Quarterly, 35(1):9--30, 1990.
• 33: James D. Herbsleb, Global Software Engineering: The Future of Socio-technical Coordination, 2007 Future of Software Engineering, p.188-198, May 23-25, 2007 [doi>10.1109/FOSE.2007.11]
• 34: James D. Herbsleb , Audris Mockus, Formulation and preliminary test of an empirical theory of coordination in software engineering, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 35: G. Hertel, S. Niedner, and S. Herrmann. Motivation of software developers in Open Source projects: an Internet-based survey of contributors to the Linux kernel. Research Policy, 32(7):1159--1177, 2003.
• 36: Pamela Hinds , Cathleen McGrath, Structures that work: social structure, work structure and coordination ease in geographically distributed teams, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180928]
• 37: A. Hintze and C. Adami. Evolution of complex modular biological networks. PloS Computational Biology, e23.eor, 2008.
• 38: Liaquat Hossain , Andrè Wu , Kenneth K S Chung, Actor centrality correlates to project based coordination, Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, November 04-08, 2006, Banff, Alberta, Canada [doi>10.1145/1180875.1180930]
• 39: H. Ibarra. Network centrality, power, and innovation involvement: Determinants of technical and administrative roles. The Academy of Management Journal, 36(3):471--501, jun 1993.
• 40: N. Kashtan and U. Alon. Spontaneous evolution of modularity and network motifs. Proceedings of the National Academy of Sciences, 102(39):13773--13778, 2005.
• 41: K. Kuwabara. Linux: A bazaar at the edge of chaos. First Monday, 5(3), March 2000.
• 42: L. Layman, L. Williams, D. Damian, and H. Bures. Essential communication practices for Extreme Programming in a global software development team. Information and Software Technology, 48(9):781--794, 2006.
• 43: L. Lopez, J. M. Gonzalez-Barahona, and G. Robles. Applying social network analysis to the information in cvs repositories. In Proceedings of the International Workshop on Mining Software Repositories, 2004.
• 44: R. Milo, N. Kashtan, S. Itzkovitz, M. E. J. Newman, and U. Alon. On the uniform generation of random graphs with prescribed degree sequences. Arxiv preprint cond-mat/0312028, 2003.
• 45: Audris Mockus , Roy T. Fielding , James Herbsleb, A case study of open source software development: the Apache server, Proceedings of the 22nd international conference on Software engineering, p.263-272, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337209]
• 46: Two case studies of open source software development: Apache and Mozilla, ACM Transactions on Software Engineering and Methodology (TOSEM), v.11 n.3, p.309-346, July 2002 [doi>10.1145/567793.567795]
• 47: Michael Molloy , Bruce Reed, A critical point for random graphs with a given degree sequence, Random Structures & Algorithms, v.6 n.2-3, p.161-179, March-May 1995
• 48: Kumiyo Nakakoji , Yasuhiro Yamamoto , Yoshiyuki Nishinaka , Kouichi Kishida , Yunwen Ye, Evolution patterns of open-source software systems and communities, Proceedings of the International Workshop on Principles of Software Evolution, May 19-20, 2002, Orlando, Florida [doi>10.1145/512035.512055]
• 49: M. E. J. Newman. Analysis of weighted networks. Physical Review E, 70:056131, 2004.
• 50: M. E. J. Newman. Finding community structure in networks using the eigenvectors of matrices. Physical Review E, 74(3):36104, 2006.
• 51: M. E. J. Newman and M. Girvan. Finding and evaluating community structure in networks. Phys. Rev. E, 69(2):026113, Feb 2004.
• 52: M. E. J. Newman, S. H. Strogatz, and D. J. Watts. Random graphs with arbitrary degree distributions and their applications. Phys. Rev. E, 64(2), Jul 2001.
• 53: D. L. Parnas, On the criteria to be used in decomposing systems into modules, Communications of the ACM, v.15 n.12, p.1053-1058, Dec. 1972 [doi>10.1145/361598.361623]
• 54: Eric S. Raymond , Tim O'Reilly, The Cathedral and the Bazaar, O'Reilly & Associates, Inc., Sebastopol, CA, 1999
• 55: M. P. Robillard. Bellairs workshop on recommender systems, 3 2008.
• 56: Manuel E. Sosa , Steven D. Eppinger , Craig M. Rowles, The Misalignment of Product Architecture and Organizational Structure in Complex Product Development, Management Science, v.50 n.12, p.1674-1689, December 2004 [doi>10.1287/mnsc.1040.0289]
• 57: M. L. Tushman and R. Katz. External communication and project performance: An investigation into the role of gatekeepers. Management Science, 26(11):1071--1085, 1980.
• 58: J. Tyler, D. Wilkinson, and B. Huberman. E-Mail as Spectroscopy: Automated Discovery of Community Structure within Organizations. The Information Society, 21(2):143--153, 2005.
• 59: Giuseppe Valetto , Mary Helander , Kate Ehrlich , Sunita Chulani , Mark Wegman , Clay Williams, Using Software Repositories to Investigate Socio-technical Congruence in Development Projects, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.25, May 20-26, 2007 [doi>10.1109/MSR.2007.33]
• 60: P. Wagstrom, J. Herbsleb, and K. Carley. A Social Network Approach To Free/Open Source Software Simulation. Proceedings of the 1st International Conference on Open Source Systems, Genova, 11th--15th July, 2005.
• 61: S. Wasserman and K. Faust. Social network analysis: Methods and applications. Cambridge University Press, 1994.
• 62: Jin Xu , Yongqin Gao , Scott Christley , Gregory Madey, A Topological Analysis of the Open Souce Software Development Community, Proceedings of the Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 7, p.198.1, January 03-06, 2005 [doi>10.1109/HICSS.2005.57]
• 63: Yunwen Ye , Yasuhiro Yamamoto , Kumiyo Nakakoji, A socio-technical framework for supporting programmers, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287674]
• 64: Jeongah Yoon , Anselm Blumer , Kyongbum Lee, An algorithm for modularity analysis of directed and weighted biological networks based on edge-betweenness centrality, Bioinformatics, v.22 n.24, p.3106-3108, December 2006 [doi>10.1093/bioinformatics/btl533]
• 65: E. Ziv, M. Middendorf, and C. Wiggins. Information-theoretic approach to network modularity. Physical Review E, 71(4):46117, 2005.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Programming teams

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Process metrics

General Terms:
Human Factors, Management, Measurement

Keywords:
collaboration, open source software, social networks