The influence of organizational structure on software quality: an empirical case study

ABSTRACT

Often software systems are developed by organizations consisting of many teams of individuals working together. Brooks states in the Mythical Man Month book that product quality is strongly affected by organization structure. Unfortunately there has been little empirical evidence to date to substantiate this assertion. In this paper we present a metric scheme to quantify organizational complexity, in relation to the product development process to identify if the metrics impact failure-proneness. In our case study, the organizational metrics when applied to data from Windows Vista were statistically significant predictors of failure-proneness. The precision and recall measures for identifying failure-prone binaries, using the organizational metrics, was significantly higher than using traditional metrics like churn, complexity, coverage, dependencies, and pre-release bug measures that have been used to date to predict failure-proneness. Our results provide empirical evidence that the organizational metrics are related to, and are effective predictors of failure-proneness.

REFERENCES

• 1: Victor R. Basili , Lionel C. Briand , Walcélio L. Melo, A Validation of Object-Oriented Design Metrics as Quality Indicators, IEEE Transactions on Software Engineering, v.22 n.10, p.751-761, October 1996 [doi>10.1109/32.544352]
• 2: V. Basili, G. Caldiera, and D. H. Rombach, "The Goal Question Metric Paradigm," in Encyclopedia of Software Engineering, Vol. 2: John Wiley and Sons, Inc., pp. 528--532, 1994.
• 3: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 4: S. Biyani , P. Santhanam, Exploring Defect Data from Development and Customer Usage on Software Modules over Multiple Releases, Proceedings of the The Ninth International Symposium on Software Reliability Engineering, p.316, November 04-07, 1998
• 5: Lionel C. Briand , Jürgen Wüst , Stefan V. Ikonomovski , Hakim Lounis, Investigating quality factors in object-oriented designs: an industrial case study, Proceedings of the 21st international conference on Software engineering, p.345-354, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302654]
• 6: Frederick P. Brooks, Jr., The mythical man-month (anniversary ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 7: S. R. Chidamber , C. F. Kemerer, A Metrics Suite for Object Oriented Design, IEEE Transactions on Software Engineering, v.20 n.6, p.476-493, June 1994 [doi>10.1109/32.295895]
• 8: M. E. Conway, "How Do Committees Invent?" Datamation, 14(4), pp. 28--31, 1968.
• 9: T. DeMarco and T. Lister, Peopleware. New York: Dorset House Publishers, 1977.
• 10: Giovanni Denaro , Mauro Pezzè, An empirical evaluation of fault-proneness models, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581371]
• 11: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 12: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 13: James D. Herbsleb , Rebecca E. Grinter, Splitting the organization and integrating the code: Conway's law revisited, Proceedings of the 21st international conference on Software engineering, p.85-95, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302455]
• 14: James D. Herbsleb , Rebecca E. Grinter, Architectures, Coordination, and Distance: Conway's Law and Beyond, IEEE Software, v.16 n.5, p.63-70, September 1999 [doi>10.1109/52.795103]
• 15: James D. Herbsleb , Audris Mockus, Formulation and preliminary test of an empirical theory of coordination in software engineering, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 16: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 17: E. J. Jackson, A User's Guide to Principal Components: John Wiley & Sons, Inc., 1991.
• 18: T. M. Khoshgoftaar , E. B. Allen , N. Goel , A. Nandi , J. McMullan, Detection of software modules with high debug code churn in a very large legacy system, Proceedings of the The Seventh International Symposium on Software Reliability Engineering (ISSRE '96), p.364, October 30-November 02, 1996
• 19: Taghi M. Khoshgoftaar , Robert M. Szabo, Improving Code Churn Predictions During the System Test and Maintenance Phases, Proceedings of the International Conference on Software Maintenance, p.58-67, September 01, 1994
• 20: D. G. Kleinbaum , L. L. Kupper , K. E. Muller, Applied regression analysis and other multivariable methods, PWS Publishing Co., Boston, MA, 1988
• 21: T. J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering, v.2 n.4, p.308-320, July 1976 [doi>10.1109/TSE.1976.233837]
• 22: Two case studies of open source software development: Apache and Mozilla, ACM Transactions on Software Engineering and Methodology (TOSEM), v.11 n.3, p.309-346, July 2002 [doi>10.1145/567793.567795]
• 23: Audris Mockus , Ping Zhang , Paul Luo Li, Predictors of customer perceived software quality, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062506]
• 24: Nachiappan Nagappan , Thomas Ball, Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study, Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, p.364-373, September 20-21, 2007 [doi>10.1109/ESEM.2007.87]
• 25: Nachiappan Nagappan , Thomas Ball, Use of relative code churn measures to predict system defect density, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062514]
• 26: Nachiappan Nagappan , Thomas Ball , Brendan Murphy, Using Historical In-Process and Product Metrics for Early Estimation of Software Failures, Proceedings of the 17th International Symposium on Software Reliability Engineering, p.62-74, November 07-10, 2006 [doi>10.1109/ISSRE.2006.50]
• 27: Nachiappan Nagappan , Thomas Ball , Andreas Zeller, Mining metrics to predict component failures, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134349]
• 28: N. Nagappan, Murphy, B., Basili, V., "The Influence of Organizational Structure On Software Quality: An Empirical Case Study," Microsoft Research Technical Report (http://research.microsoft.com), MSR-TR-2008-11, 2008.
• 29: L. Osterweil, Software processes are software too, Proceedings of the 9th international conference on Software Engineering, p.2-13, March 1987, Monterey, California, United States
• 30: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 31: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Where the bugs are, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 32: D. L. Parnas, On the criteria to be used in decomposing systems into modules, Communications of the ACM, v.15 n.12, p.1053-1058, Dec. 1972 [doi>10.1145/361598.361623]
• 33: Dewayne E. Perry , Nancy Staudenmayer , Lawrence G. Votta, People, Organizations, and Process Improvement, IEEE Software, v.11 n.4, p.36-45, July 1994 [doi>10.1109/52.300082]
• 34: A. Podgurski , L. A. Clarke, A Formal Model of Program Dependences and its Implications for Software Testing, Debugging, and Maintenance, IEEE Transactions on Software Engineering, v.16 n.9, p.965-979, September 1990 [doi>10.1109/32.58784]
• 35: Adrian Schröter , Thomas Zimmermann , Andreas Zeller, Predicting component failures at design time, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159739]
• 36: Hongyu Zhang , Xiuzhen Zhang, Comments on "Data Mining Static Code Attributes to Learn Defect Predictors", IEEE Transactions on Software Engineering, v.33 n.9, p.635-637, September 2007 [doi>10.1109/TSE.2007.70706]
• 37: Thomas Zimmermann , Peter Weissgerber , Stephan Diehl , Andreas Zeller, Mining Version Histories to Guide Software Changes, IEEE Transactions on Software Engineering, v.31 n.6, p.429-445, June 2005 [doi>10.1109/TSE.2005.72]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics; Performance measures; Process metrics

General Terms:
Human Factors, Measurement, Reliability

Keywords:
code churn, developers, empirical studies, failures, organizational structure, software mining