The impact of time controlled reading on software inspection effectiveness and efficiency: a controlled experiment

ABSTRACT

Reading techniques help to guide reviewers during individual software inspections. In this experiment, we completely transfer the principle of statistical usage testing to inspection reading techniques for the first time. Statistical usage testing relies on a usage profile to determine how intensively certain parts of the system shall be tested from the users' perspective. Usage-based reading applies statistical usage testing principles by utilizing prioritized use cases as a driver for inspecting software artifacts (e.g., design). In order to reflect how intensively certain use cases should be inspected, time budgets are introduced to usage-based reading where a maximum inspection time is assigned to each use case. High priority use cases receive more time than low priority use cases. A controlled experiment is conducted with 23 Software Engineering M.Sc. students inspecting a design document. In this experiment, usage-based reading without time budgets is compared with time controlled usage-based reading. The result of the experiment is that time budgets do not significantly improve inspection performance. In conclusion, it is sufficient to only use prioritized use cases to successfully transfer statistical usage testing to inspections.

REFERENCES

• 1: B. Anda, H. C. Benestad, and S. E. Hove. A multiple-case study of software effort estimation based on use case points. In Proc. of the fourth International Symposium on Empirical Software Engineering (ISESE 2005), pages 407--416, 2005.
• 2: Carina Andersson , Thomas Thelin , Per Runeson , Nina Dzamashvili, An Experimental Evaluation of Inspection and Testing for Detection of Design Faults, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.174, September 30-October 01, 2003
• 3: A. Aurum, H. Petersson, and C.Wohlin. State-of-the-art: software inspections after 25 years. Softw. Test., Verif. Reliab., 12(3):133--154, 2002.
• 4: Victor R. Basili , Scott Green , Oliver Laitenberger , Forrest Shull , Sivert Sørumgård , Marvin V. Zelkowitz, The empirical investigation of perspective-based reading, University of Maryland at College Park, College Park, MD, 1995
• 5: David B. Bisant , James R. Lyle, A Two-Person Inspection Method to Improve Programming Productivity, IEEE Transactions on Software Engineering, v.15 n.10, p.1294-1304, October 1989 [doi>10.1109/TSE.1989.559782]
• 6: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 7: M. E. Fagan. Design and code inspections to reduce errors in program development. IBM Systems Journal, 15(3):182--211, 1976.
• 8: Philip M. Johnson, An instrumented approach to improving software quality through formal technical review, Proceedings of the 16th international conference on Software engineering, p.113-122, May 16-21, 1994, Sorrento, Italy
• 9: Joachim Karlsson , Kevin Ryan, A Cost-Value Approach for Prioritizing Requirements, IEEE Software, v.14 n.5, p.67-74, September 1997 [doi>10.1109/52.605933]
• 10: Shinji Kusumoto , Fumikazu Matukawa , Katsuro Inoue , Shigeo Hanabusa , Yuusuke Maegawa, Estimating Effort by Use Case Points: Method, Tool and Case Study, Proceedings of the Software Metrics, 10th International Symposium, p.292-299, September 11-17, 2004 [doi>10.1109/METRICS.2004.21]
• 11: Oliver Laitenberger , Colin Atkinson , Maud Schlich , Khaled El Emam, An experimental comparison of reading techniques for defect detection in UML design documents, Journal of Systems and Software, v.53 n.2, p.183-204, Aug. 31, 2000 [doi>10.1016/S0164-1212(00)00052-2]
• 12: Oliver Laitenberger , Khaled El Emam , Thomas G. Harbich, An Internally Replicated Quasi-Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents, IEEE Transactions on Software Engineering, v.27 n.5, p.387-421, May 2001 [doi>10.1109/32.922713]
• 13: Soren Lauesen, Software Requirements: Styles and Techniques, Pearson Education, 2001
• 14: Dean Leffingwell , Don Widrig, Managing Software Requirements: A Use Case Approach, Pearson Education, 2003
• 15: Richard C. Linger, Cleanroom Process Model, IEEE Software, v.11 n.2, p.50-58, March 1994 [doi>10.1109/52.268956]
• 16: Johnny Martin , W. T. Tsai, N-Fold inspection: a requirements analysis technique, Communications of the ACM, v.33 n.2, p.225-232, Feb. 1990 [doi>10.1145/75577.75587]
• 17: Vahid Mashayekhi , Janet M. Drake , Wei-Tek Tsai , John Riedl, Distributed, Collaborative Software Inspection, IEEE Software, v.10 n.5, p.66-75, September 1993 [doi>10.1109/52.232404]
• 18: John D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper, Authorhouse, 2004
• 19: Ian Sommerville, Software Engineering: (Update) (8th Edition) (International Computer Science), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2006
• 20: Thomas Thelin , Carina Andersson , Per Runeson , Nina Dzamashvili-Fogelstrom, A Replicated Experiment of Usage-Based and Checklist-Based Reading, Proceedings of the Software Metrics, 10th International Symposium, p.246-256, September 11-17, 2004 [doi>10.1109/METRICS.2004.3]
• 21: T. Thelin, P. Runeson, and B. Regnell. Usage-based reading an experiment to guide reviewers with use cases. Information & Software Technology, 43(15):925--938, 2001.
• 22: Thomas Thelin , Per Runeson , Claes Wohlin, An Experimental Comparison of Usage-Based and Checklist-Based Reading, IEEE Transactions on Software Engineering, v.29 n.8, p.687-704, August 2003 [doi>10.1109/TSE.2003.1223644]
• 23: Thomas Thelin , Per Runeson , Claes Wohlin, Prioritized Use Cases as a Vehicle for Software Inspections, IEEE Software, v.20 n.4, p.30-33, July 2003 [doi>10.1109/MS.2003.1207451]
• 24: Thomas Thelin , Per Runeson , Claes Wohlin , Thomas Olsson , Carina Andersson, How much Information is Needed for Usage-Based Reading? " A Series of Experiments, Proceedings of the 2002 International Symposium on Empirical Software Engineering, p.127, October 03-04, 2002
• 25: Lawrence G. Votta, Jr., Does every inspection need a meeting?, Proceedings of the 1st ACM SIGSOFT symposium on Foundations of software engineering, p.107-114, December 08-10, 1993, Los Angeles, California, United States
• 26: Gwendolyn H. Walton , J. H. Poore , Carmen J. Trammell, Statistical testing of software based on a usage model, Software—Practice & Experience, v.25 n.1, p.97-108, Jan. 1995 [doi>10.1002/spe.4380250106]
• 27: Dietmar Winkler , Michael Halling , Stefan Biffl, Investigating the Effect of Expert Ranking of Use Cases for Design Inspection, Proceedings of the 30th EUROMICRO Conference, p.362-371, August 31-September 03, 2004 [doi>10.1109/EUROMICRO.2004.49]
• 28: C. Wohlin, B. Regnell, A. Wesslén, and H. Cosmo. User-centered software engineering: A comprehensive view of software development. In Proc. of the Nordic Seminar on Dependable Computing Systems, pages 229--240, 1994.
• 29: Claes Wohlin , Per Runeson, Defect content estimations from review data, Proceedings of the 20th international conference on Software engineering, p.400-409, April 19-25, 1998, Kyoto, Japan
• 30: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000
• 31: I.-T. Z.120. Message sequence charts, msc, itu-t recommendation z.120, 1996.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Code inspections and walk-throughs

General Terms:
Experimentation

Keywords:
effectiveness, efficiency, experiment, software inspection, time-controlled usage-based reading