Maximising the information gained from an experimental analysis of code inspection and static analysis for concurrent java components

ABSTRACT

The results of empirical studies are limited to particular contexts, difficult to generalise and the studies themselves are expensive to perform. Despite these problems, empirical studies in software engineering can be made effective and they are important to both researchers and practitioners. The key to their effectiveness lies in the maximisation of the information that can be gained by examining existing studies, conducting power analyses for an accurate minimum sample size and benefiting from previous studies through replication. This approach was applied in a controlled experiment examining the combination of automated static analysis tools and code inspection in the context of verification and validation (V&V) of concurrent Java components. The combination of these V&V technologies was shown to be cost-effective despite the size of the study, which thus contributes to research in V&V technology evaluation.

REFERENCES

• 1: Gregory R. Andrews, Concurrent programming: principles and practice, Benjamin-Cummings Publishing Co., Inc., Redwood City, CA, 1991
• 2: Artho, C. Finding faults in multi-threaded programs, Federal Institute of Technology, Zurich-Austin, 2001.
• 3: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 4: V R Basili , R W Selby , D H Hutchens, Experimentation in software engineering, IEEE Transactions on Software Engineering, v.12 n.7, p.733-743, July 1986
• 5: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 6: Jeff Carver , John Van Voorhis , Victor Basili, Understanding the Impact of Assumptions on Experimental Validity, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.251-260, August 19-20, 2004 [doi>10.1109/ISESE.2004.32]
• 7: Richard H. Carver , Kuo-Chung Tai, Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs, IEEE Transactions on Software Engineering, v.24 n.6, p.471-490, June 1998 [doi>10.1109/32.689403]
• 8: Coakes, S.J. and Steed, L.G. SPSS: Analysis without Anguish (Version 10.0 for Windows). John Wiley and Sons, 2001.
• 9: Cohen, J. Statistical Power Analysis for the Behavioural Sciences. Lawrence Erlbaum Associates, 1988.
• 10: James C. Corbett, Evaluating Deadlock Detection Methods for Concurrent Software, IEEE Transactions on Software Engineering, v.22 n.3, p.161-180, March 1996 [doi>10.1109/32.489078]
• 11: Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact, Empirical Software Engineering, v.10 n.4, p.405-435, October 2005 [doi>10.1007/s10664-005-3861-2]
• 12: Dyba, T., Kampenes, V.B. and Sjøberg, D.I.K. A systematic review of statistical power in software engineering experiments. Journal of Information and Software Technology, 48, 8 (2006), 745--755.
• 13: Yaniv Eytani , Klaus Havelund , Scott D. Stoller , Shmuel Ur, Towards a framework and a benchmark for testing tools for multi-threaded programs: Research Articles, Concurrency and Computation: Practice & Experience, v.19 n.3, p.267-279, March 2007 [doi>10.1002/cpe.v19:3]
• 14: Cormac Flanagan , Stephen N Freund, Atomizer: a dynamic atomicity checker for multithreaded programs, Proceedings of the 31st ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.256-267, January 14-16, 2004, Venice, Italy
• 15: H. H. Hallal , E. Alikacem , W. P. Tunney , S. Boroday , A. Petrenko, Antipattern-Based Detection of Deficiencies in Java Multithreaded Software, Proceedings of the Quality Software, Fourth International Conference on (QSIC'04), p.258-267, September 08-10, 2004 [doi>10.1109/QSIC.2004.8]
• 16: William C. Hetzel, An experimental analysis of program verification methods., 1976
• 17: Hovemeyer, D. and Pugh, W., Finding Concurrency bugs in Java. In Proceedings of the 23rd Annual ACM SIGACTSIGOPS Symposium on Principles of Distributed Computing (PODC 2004) Workshop on Concurrency and Programs, (2004).
• 18: Howell, D.C. Statistical Methods for Psychology. Wadsworth Publishing Compnay, 1997.
• 19: Andreas Jedlitschka , Marcus Ciolkowski, Towards Evidence in Software Engineering, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.261-270, August 19-20, 2004 [doi>10.1109/ISESE.2004.31]
• 20: Jedlitschka, A. and Pfahl, D., Reporting Guidelines for Controlled Experiments in Software Engineering. In Proceedings of the 2005 International Symposium on Empirical Software Engineering (ISESE'05), (2005), 95--104.
• 21: Ross Jeffery , Louise Scott, Has Twenty-five Years of Empirical Software Engineering Made a Difference?, Proceedings of the Ninth Asia-Pacific Software Engineering Conference, p.539, December 04-06, 2002
• 22: N. Juristo , A. M. Moreno , S. Vegas, A Survey on Testing Technique Empirical Studies: How Limited is our Knowledge, Proceedings of the 2002 International Symposium on Empirical Software Engineering, p.161, October 03-04, 2002
• 23: Erik Kamsties , Christopher M. Lott, An Empirical Evaluation of Three Defect-Detection Techniques, Proceedings of the 5th European Software Engineering Conference, p.362-383, September 25-28, 1995
• 24: Kitchenham, B.A., The Case Against Software Benchmarking, Keynote Lecture. In Proceedings of The European Software Measurement Conference (FESMA-DASMA 2001), (2001), 1--9.
• 25: Kitchenham, B.A. Procedures for Performing Systematic Reviews, Keele University, 2004.
• 26: Kitchenham, B.A., Linkman, S.G. and Fry, J.S., Experimenter induced distortions in empirical software engineering. In Proceedings of 2nd International Workshop on Empirical Software Engineering (WSESE), (2003), 7--15.
• 27: Barbara Kitchenham , Shari Lawrence Pfleeger , Norman Fenton, Towards a Framework for Software Measurement Validation, IEEE Transactions on Software Engineering, v.21 n.12, p.929-944, December 1995 [doi>10.1109/32.489070]
• 28: Barbara A. Kitchenham , Shari Lawrence Pfleeger , Lesley M. Pickard , Peter W. Jones , David C. Hoaglin , Khaled El Emam , Jarrett Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, v.28 n.8, p.721-734, August 2002 [doi>10.1109/TSE.2002.1027796]
• 29: Lea, D. Overview of package util.concurrent Release 1.3.4. Available online at http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html.
• 30: Lindsay, R.M. and Ehrendberg, A.S.C. The Design of Replicated Studies. The American Statistician, 47, 3 (1993), 217--228.
• 31: Long, B., Duke, R., Goldson, D., Strooper, P. and Wildman, L., Mutation-Based Evaluation of a Method for Verifying Concurrent Java Components. In Proceedings of the 2nd International Workshop on Parallel and Distributed Systems: Testing and Debugging (PADTAD), (2004).
• 32: Long, B., Strooper, P. and Hoffman, D. Tool Support for Testing Concurrent Java Components. IEEE Transactions in Software Engineering, 29, 6 (2003), 555--566.
• 33: Brad Long , Paul Strooper , Luke Wildman, A method for verifying concurrent Java components based on an analysis of concurrency failures: Research Articles, Concurrency and Computation: Practice & Experience, v.19 n.3, p.281-294, March 2007 [doi>10.1002/cpe.v19:3]
• 34: Lott, C.M. Comparing Reading and Testing Techniques, Available online at: http://www.chris-lott.org/work/exp/.
• 35: Lott, C.M. and Rombach, H.D. Repeatable Software Engineering Experiments for Comparing Defect-Detection Techniques. Empirical Software Engineering, 1, 3 (1996), 241--277.
• 36: Jeff Magee , Jeff Kramer, Concurrency: state models & Java programs, John Wiley & Sons, Inc., New York, NY, 2000
• 37: Miller, I. and Freund, J.E. Probability and Statistics for Engineers. Prentice-Hall, 1965.
• 38: Miller, J., Daly, J., Wood, M., Roper, M. and Brooks, A. Statistical power and its subcomponents - missing and misunderstood concepts in software engineering empirical research. Journal of Information and Software Technology, 39, 4 (1997), 285--295.
• 39: Glenford J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Communications of the ACM, v.21 n.9, p.760-768, Sept. 1978 [doi>10.1145/359588.359602]
• 40: Ernesto Novillo , Paul Lu, A Case Study of Selected SPLASH-2 Applications and the SBT Debugging Tool, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.290.2, April 22-26, 2003
• 41: David Lorge Parnas, The Limits of Empirical Studies of Software Engineering, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.2, September 30-October 01, 2003
• 42: Pickard, L.M., Kitchenham, B.A. and Jones, P. Combining Empirical Results in Software Engineering. Information and Software Technology, 40, 14 (1998), 811--821.
• 43: Glen W. Russell, Experience With Inspection in Ultralarge-Scale Development, IEEE Software, v.8 n.1, p.25-31, January 1991 [doi>10.1109/52.62929]
• 44: Selby, R.W., Combining Software Testing Strategies: An Empirical Evaluation. In Proceedings of the ACM/SIGSOFT IEEE Workshop on Software Testing, (1986), 82--90.
• 45: Forrest Shull , Victor Basili , Jeffrey Carver , José C. Maldonado , Guilherme Horta Travassos , Manoel Mendonça , Sandra Fabbri, Replicating Software Engineering Experiments: Addressing the Tacit Knowledge Problem, Proceedings of the 2002 International Symposium on Empirical Software Engineering, p.7, October 03-04, 2002
• 46: Dag I. K. Sjoberg , Jo E. Hannay , Ove Hansen , Vigdis By Kampenes , Amela Karahasanovic , Nils-Kristian Liborg , Anette C. Rekdal, A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, v.31 n.9, p.733-753, September 2005 [doi>10.1109/TSE.2005.97]
• 47: Clemens Szyperski, Component software: beyond object-oriented programming, ACM Press/Addison-Wesley Publishing Co., New York, NY, 1998
• 48: Walter F. Tichy, Should Computer Scientists Experiment More?, Computer, v.31 n.5, p.32-40, May 1998 [doi>10.1109/2.675631]
• 49: Ur, S. and Eytani, Y., Compiling a Benchmark of Documented Multi-threaded Bugs. In Proceedings of the 2nd International Workshop on Parallel and Distributed Systems: Testing and Debugging (PADTAD), (2004).
• 50: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000
• 51: Margaret A. Wojcicki , Paul Strooper, A state-of-practice questionnaire on verification and validation for concurrent programs, Proceeding of the 2006 workshop on Parallel and distributed systems: testing and debugging, July 17-17, 2006, Portland, Maine, USA [doi>10.1145/1147403.1147405]
• 52: Wood, M., Roper, M., Brooks, A. and Miller, J., Comparing and Combining Software Defect Detection Techniques: A Replicated Empirical Study. In Proceedings of the 6th European Software Engineering Conference, (1997), 262--277.
• 53: Zar, J.H. Significance Testing of the Spearman Rank Correlation Coefficient. Journal of the American Statistical Association, 67, 339 (1972), 578--580.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Validation

General Terms:
Experimentation, Verification

Keywords:
concurrent Java components, controlled experiment, verification and validation