Impact of research on practice in the field of inspections, reviews and walkthroughs: learning from successful industrial uses

ABSTRACT

Software inspections, reviews, and walkthroughs have become a standard process component in many software development domains. Maturity level 3 of the CMM-I requires establishment of peer reviews [12] and substantial sustained improvements in quality and productivity have been reported as a result of using reviews ([16], [21], [22], [27]). The NSF Impact project identifies the degree to which these industrial success cases have been instigated and improved by research in software engineering.

This research identifies that there is widespread adoption of inspections, reviews or walkthroughs but that companies do not generally exploit their full potential. However there exist sustained industrial success cases with respect to the wide-spread and measurably successful application of them. It also identifies research in software engineering that can be credibly documented as having influenced the industrial success cases. Credible documentation may exist in the form of publications or documented reports by witnesses. Due to the semi-formal nature of inspections, reviews, and walkthroughs, a specific focus is given to empirical research results as motivators for adoption. Through the examination of one detailed case study, it is shown that software engineering research has had a significant impact on practice and that the impact can be traced in this case from research to that practice. The case study chosen provides evidence of both success and failure regarding sustained application in practice.

Thus the analysis of historic impact chains of research reveals a clear impact of software engineering research on sustained industrial success for inspections, reviews and walkthroughs. More importantly, in impact chains where the empirical results have not been established, we conclude that success has not been achieved or has not been sustained.

The paper closes with (1) lessons learned for creating the sustained use and impact of semi-formal software engineering processes, (2) a request for researchers and practitioners to further consider how their work can improve the effectiveness of research and practice, and (3) a request to contribute additional success cases and impact factors to the authors database for future enhancements of this paper.

REFERENCES

• 1: Aurum, A. and Petersson, H. and Wohlin, C., State-of-the-art: software inspections after 25 years, Journal of Software Testing, Verification and Reliability, 12(3), Sept. 2002, pp. 133--54.
• 2: Kent Beck, Extreme programming explained: embrace change, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 3: Victor R. Basili, Evolving and packaging reading technologies, Journal of Systems and Software, v.38 n.1, p.3-12, July 1997 [doi>10.1016/S0164-1212(97)00065-4]
• 4: Basili, V.R., Caldiera, G., and Rombach, H.D., "Goal Question Metric Paradigm," in Encyclopedia of Software Engineering, vol. 1, J.J. Marciniak, Ed., 1994, pp. 528--532.
• 5: Basili, V., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sorumgard, S., and Zelkowitz, M., 1996. The Empirical Investigation of Perspective-based Reading. Journal of Empirical Software Engineering, 2(1):133164.
• 6: Victor R. Basili , Frank E. McGarry , Rose Pajerski , Marvin V. Zelkowitz, Lessons learned from 25 years of process improvement: the rise and fall of the NASA software engineering laboratory, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581351]
• 7: Victor R. Basili , Richard W. Selby, Comparing the Effectiveness of Software Testing Strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1987 [doi>10.1109/TSE.1987.232881]
• 8: Barry Boehm , Victor R. Basili, Software Defect Reduction Top 10 List, Computer, v.34 n.1, p.135-137, January 2001 [doi>10.1109/2.962984]
• 9: L. Briand et al., Quality Assurance Technologies for the EURO Conversion--Industrial Experience at Allianz Life Assurance, Proc. 2nd Int'l Software Quality Week Europe, Software Research Inc., 1998, pp. 1--23.
• 10: Ciolkowski, M., Shull, F., and Biffl, S.: Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering, ESERNET Method and Experience, R. Conradi, ed., LNCS, Springer-Verlag, 2003, Volume 2765 / 2003, August 2003, pp. 104--128.
• 11: Marcus Ciolkowski , Oliver Laitenberger , Stefan Biffl, Software Reviews: The State of the Practice, IEEE Software, v.20 n.6, p.46-51, November 2003 [doi>10.1109/MS.2003.1241366]
• 12: CMMI Product Team, Software Engineering Institute, Capability Maturity Model® Integration (CMMISM), Version 1.1, CMU/SEI-2002-TR-012.
• 13: Deming, W.: Out of the crisis, SPC Press, 1982.
• 14: Endres, A., Rombach, H.D., A Handbook of Software and Systems Engineering - Empirical Observations, Laws and Theories, Edison-Wesley, 2003.
• 15: Jacky Estublier , David Leblang , André van der Hoek , Reidar Conradi , Geoffrey Clemm , Walter Tichy , Darcy Wiborg-Weber, Impact of software engineering research on the practice of software configuration management, ACM Transactions on Software Engineering and Methodology (TOSEM), v.14 n.4, p.383-430, October 2005 [doi>10.1145/1101815.1101817]
• 16: Fagan, M.E.: Design and Code Inspections to Reduce Errors in Program Development, IBM Systems J., vol. 15, no. 3, 1976, pp. 182--211.
• 17: Tom Gilb , Dorothy Graham , Susannah Finzi, Software Inspection, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1993
• 18: William C. Hetzel, An experimental analysis of program verification methods., The University of North Carolina at Chapel Hill, 1976
• 19: IEEE Std. 1028-1997 (1998): Standard for Software Reviews, The Institute of Electrical and Electronics Engineering, Inc. ISBN 1-55937-987-1
• 20: Kohli, Robert, O., and Radice, R.A., Low Level Design Inspection Specification, TR 21.629, IBM System Communications Division, Kingston NY 12401.
• 21: Oliver Laitenberger , Jean-Marc DeBaud, An encompassing life cycle centric survey of software inspection, Journal of Systems and Software, v.50 n.1, p.5-31, Jan. 2000 [doi>10.1016/S0164-1212(99)00073-4]
• 22: Oliver Laitenberger , Marek Leszak , Dieter Stoll , Khaled El Emam, Quantitative Modeling of Software Reviews in an Industrial Setting, Proceedings of the 6th International Symposium on Software Metrics, p.312, November 04-06, 1999
• 23: O. Laitenberger, S. Vegas, and M. Ciolkowski, The State of the Practice of Review and Review Technologies in Germany, VISEK technical report 011.02, 2002.
• 24: Richard C. Linger , Bernard I. Witt , H. D. Mills, Structured Programming; Theory and Practice the Systems Programming Series, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1979
• 25: Lott, C.M. and Rombach, H.D., "Repeatable Software Engineering Experiments for Comparing Defect-Detection Techniques," 1996.
• 26: Marciniak, J.J., 1994. Reviews and Audits. In: Marciniak, J.J., editor, Encyclopedia of Software Engineering, volume 2, pages 10841090. John Wiley and Sons.
• 27: Frank McGarry , Bill Decker, Attaining Level 5 in CMM Process Maturity, IEEE Software, v.19 n.6, p.87-96, November 2002
• 28: Harlan D. Mills, Stepwise Refinement and Verification in Box-Structured Systems, Computer, v.21 n.6, p.23-36, June 1988 [doi>10.1109/2.948]
• 29: Mills, H: "Cleanroom Engineering", American Programmer, Pages 31--37, May 1991.
• 30: Glenford J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Communications of the ACM, v.21 n.9, p.760-768, Sept. 1978 [doi>10.1145/359588.359602]
• 31: National Institute of Standards and Technology, The Economic Impact of Inadequate Infrastructure for Software Testing, http://www.nist.gov/public_affairs/releases/n02-10.htm, May 2002
• 32: Leon J. Osterweil , Lori A. Clarke , Michael Evangelist , Jeffrey Kramer , Dieter Rombach , Alexander L. Wolf, The impact project (panel session): determining the impact of software engineering research upon practice, ACM SIGSOFT Software Engineering Notes, v.25 n.6, p.108-109, Nov. 2000
• 33: Barbara G. Ryder , Mary Lou Soffa , Margaret Burnett, The impact of software engineering research on modern progamming languages, ACM Transactions on Software Engineering and Methodology (TOSEM), v.14 n.4, p.431-477, October 2005 [doi>10.1145/1101815.1101818]
• 34: Richard W. Selby , Victor R. Basili , F. Terry Baker, Cleanroom Software Development: An Empirical Evaluation, IEEE Transactions on Software Engineering, v.13 n.9, p.1027-1037, September 1987 [doi>10.1109/TSE.1987.233525]
• 35: Solingen, R. van, Berghout, E., The Goal/Question/Metric Method. A Practical Guide for Quality Improvement of Software Development, London: McGraw-Hill, 1999, ISBN: 0-07-709553-7.
• 36: Robert G. Ebenau , Susan H. Strauss, Software inspection process, McGraw-Hill, Inc., New York, NY, 1993
• 37: Lawrence G. Votta, Jr., Does every inspection need a meeting?, ACM SIGSOFT Software Engineering Notes, v.18 n.5, p.107-114, Dec. 1993
• 38: Weinberg, G.M. and Freedman, D.P., 1984. Reviews, Walkthroughs, and Inspections. IEEE Transactions on Software Engineering, 12(1):6872.
• 39: Peer reviews in software: a practical guide, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 40: Edward Yourdon, Structured walkthroughs: 4th edition, Yourdon Press, Upper Saddle River, NJ, 1989

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Code inspections and walk-throughs

Keywords:
cause-effects for successful innovation, empirical evidence, innovation, inspections, research, reviews, technology transfer, walkthroughs