PBR vs. checklist: a replication in the n-fold inspection context

ABSTRACT

Inspection is considered a powerful method to check software documents for defects. Many published work shows that inspections in requirements specification phase are particularly effective and efficient. Perspective-Based Reading (PBR) is one of the systematic techniques to support defect detection in requirements documents. In this paper we describe an experiment to validate the effectiveness of PBR in a meeting-based N-fold inspection. Our goals were: (1) re-test the hypothesis of the original experiment that PBR helps to increase individual and team defect detection effectiveness compared to an checklist approach; (2) investigate the different impact of PBR and checklist on the effectiveness of N-fold team meeting; and (3) investigate some interesting characteristics of PBR (e.g. the relationship between background experiences and performance of the subjects). The results of the study showed that PBR was significantly more effective than checklist (supporting the original study). We also found that the team meeting is much more important for checklist teams, based on the number of meeting gains and the number of false defects eliminated. Finally, we found that teams using the PBR techniques have less overlap in their defect detection than those using checklist. The ultimate goal is to provide best practices (guidance) for applying PBR in software inspection and also some advice for PBR (or software inspections) process improvement.

REFERENCES

• 1: Ballman, K., and Votta, L.G. Organizational congestion in large-scale software development. In Proceeding of the Third International Conference on the Software Process. IEEE CS Press, Los Alamitos, CA, 1994, 123--134.
• 2: Basili, V. R., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sørumgård, S., and Zelkowitz, M. V. The Empirical Investigation of Perspective-Based Reading. Empirical Software Engineering: An International Journal, 1, 2 (1996), 133--164.
• 3: V R Basili , R W Selby , D H Hutchens, Experimentation in software engineering, IEEE Transactions on Software Engineering, v.12 n.7, p.733-743, July 1986
• 4: Ciolkowski, M., Differding, C., Laitenberger, O., and Munch, J. Empirical Investigation of Perspective-Based Reading: A Replicated Experiment, Technical Report ISERN-97-13, Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany, 1997.
• 5: Fagan, M. Design and code inspections to reduce errors in program development. IBM System Journal, 15, 3 (1976), 182--211.
• 6: Daniel P. Freedman , Gerald M. Weinberg, Handbook of Walkthroughs, Inspections, and Technical Reviews: Evaluating Programs, Projects, and Products, Dorset House Publishing Co., Inc., New York, NY, 2000
• 7: Tom Gilb , Dorothy Graham , Susannah Finzi, Software Inspection, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1993
• 8: Watts S. Humphrey, A Discipline for Software Engineering, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 9: Philip M. Johnson , Danu Tjahjono, Assessing software review meetings: a controlled experimental study using CSRS, Proceedings of the 19th international conference on Software engineering, p.118-127, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253252]
• 10: Erik Kamsties , Christopher M. Lott, An Empirical Evaluation of Three Defect-Detection Techniques, Proceedings of the 5th European Software Engineering Conference, p.362-383, September 25-28, 1995
• 11: John C. Knight , E. Ann Myers, An improved inspection technique, Communications of the ACM, v.36 n.11, p.51-61, Nov. 1993 [doi>10.1145/163359.163366]
• 12: Laitenberger, O., and DeBaud, J. An Encompassing Life-Cycle Centric Survey of Software Inspection. Technical Report ISERN-98-32, Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany, 1998.
• 13: Lanubile, F. and Visaggio, G. Evaluating Defect Detection Techniques for Software Requirements Inspections, Technical Report ISERN-00-08, International Software Engineering Research Network.
• 14: Johnny Martin , W. T. Tsai, N-Fold inspection: a requirements analysis technique, Communications of the ACM, v.33 n.2, p.225-232, Feb. 1990 [doi>10.1145/75577.75587]
• 15: P. McCarthy , A. Porter , H. Siy , L. G. Votta, Jr., An experiment to assess cost-benefits of inspection meetings and their alternatives: a pilot study, Proceedings of the 3rd International Symposium on Software Metrics: From Measurement to Empirical Results, p.100, March 25-26, 1996
• 16: J. M. Perpich , D. E. Perry , A. A. Porter , L. G. Votta , M. W. Wade, Anywhere, anytime code inspections: using the Web to remove inspection bottlenecks in large-scale software development, Proceedings of the 19th international conference on Software engineering, p.14-21, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253234]
• 17: Adam A. Porter , Philip M. Johnson, Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies, IEEE Transactions on Software Engineering, v.23 n.3, p.129-145, March 1997 [doi>10.1109/32.585501]
• 18: Adam Porter , Lawrence Votta, Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects, Empirical Software Engineering, v.3 n.4, p.355-379, December 1998 [doi>10.1023/A:1009776104355]
• 19: Adam A. Porter , Lawrence G. Votta, Jr. , Victor R. Basili, Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, IEEE Transactions on Software Engineering, v.21 n.6, p.563-575, June 1995 [doi>10.1109/32.391380]
• 20: Glen W. Russell, Experience With Inspection in Ultralarge-Scale Development, IEEE Software, v.8 n.1, p.25-31, January 1991 [doi>10.1109/52.62929]
• 21: Bordin Sapsomboon , Michael Spring, Shared defect detection: the effects of annotations in asynchronous software inspection, 2000
• 22: G. Michael Schneider , Johnny Martin , W. T. Tsai, An experimental study of fault detection in user requirements documents, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.2, p.188-204, April 1992 [doi>10.1145/128894.128897]
• 23: Forrest Joseph Shull , Victor R. Basili, Developing techniques for using software documents: a series of empirical studies, 1998
• 24: Forrest Shull , Jeffrey Carver , Guilherme H. Travassos, An empirical methodology for introducing software processes, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, September 10-14, 2001, Vienna, Austria
• 25: Forrest Shull , Manoel G. Mendoncça , Victor Basili , Jeffrey Carver , José C. Maldonado , Sandra Fabbri , Guilherme Horta Travassos , Maria Cristina Ferreira, Knowledge-Sharing Issues in Experimental Software Engineering, Empirical Software Engineering, v.9 n.1-2, p.111-137, March 2004 [doi>10.1023/B:EMSE.0000013516.80487.33]
• 26: Forrest Shull , Ioana Rus , Victor Basili, How Perspective-Based Reading Can Improve Requirements Inspections, Computer, v.33 n.7, p.73-79, July 2000 [doi>10.1109/2.869376]
• 27: Michael Stein , John Riedl , Sören J. Harner , Vahid Mashayekhi, A case study of distributed, asynchronous software inspection, Proceedings of the 19th international conference on Software engineering, p.107-117, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253250]
• 28: Thompsom, C., & Riedl, J. (1995). Collaborative asynchronous inspection of software using Lotus Notes. Technical Report 95-047, Computer Science Department, University of Minnesota.
• 29: Tripp, L., Struck, W., and Pflung, B. The application of multiple team inspections on a safety-critical software standard. Proceeding of 4th Software Engineering Standards Application Workshop. IEEE CS Press, Los Alamitos, CA, 1991, 106--111
• 30: Lawrence G. Votta, Jr., Does every inspection need a meeting?, Proceedings of the 1st ACM SIGSOFT symposium on Foundations of software engineering, p.107-114, December 08-10, 1993, Los Angeles, California, United States
• 31: Wheeler, D. A., Brykczynski, B, and Meeson, R. N. Software Inspection: An Industry Best Practice. IEEE CS Press, Los Alamitos, CA, 1996.
• 32: Zhijun Zhang , Victor Basili , Ben Shneiderman, Perspective-based Usability Inspection: An Empirical Validationof Efficacy, Empirical Software Engineering, v.4 n.1, p.43-69, March 1999 [doi>10.1023/A:1009803214692]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Productivity

General Terms:
Documentation, Experimentation, Human Factors, Management, Measurement, Performance, Verification

Keywords:
PBR, experimentation, human subjects, n-fold inspection