A state-of-practice questionnaire on verification and validation for concurrent programs

ABSTRACT

Research in verification and validation (V&V) for concurrent programs can be guided by practitioner information. A survey was therefore run to gain state-of-practice information in this context. The survey presented in this paper collected state-of-practice information on V&V technology in concurrency from 35 respondents. The results of the survey can help refine existing V&V technology by providing a better understanding of the context of V&V technology usage. Responses to questions regarding the motivation for selecting V&V technologies can help refine a systematic approach to V&V technology selection.

REFERENCES

• 1: Concurrency Interest Mailing List, Available online at: http://altair.cs.oswego.edu/mailman/listinfo/concurrency-interest.
• 2: developerWorks : Java technology : Forums : Multithreaded Java programming, Available online at: http://www-128.ibm.com/developerworks/forums/dw_forum.jsp?forum=176&cat=10.
• 3: FindBugs™ - Find Bugs in Java Programs, Available online at: http://findbugs.sourceforge.net/.
• 4: JUnit, Testing Resources for Extreme Programming, Available online at: http://www.junit.org/index.htm.
• 5: Macker, Available online at: http://innig.net/macker/.
• 6: PMD, Available online at: http://pmd.sourceforge.net/.
• 7: The SMV System, Available online at: http://www.cs.cmu.edu/~modelcheck/smv.html.
• 8: Spin - Formal Verification, Available online at: http://spinroot.com/spin/whatispin.html.
• 9: Carina Andersson , Per Runeson, Verification and Validation in Industry " A Qualitative Survey on the State of Practice, Proceedings of the 2002 International Symposium on Empirical Software Engineering, p.37, October 03-04, 2002
• 10: Bell Laboratories and Lucent Technologies. VeriSoft, Available online at: http://cm.bell-labs.com/who/god/verisoft/.
• 11: Burn, O. Checkstyle, Available online at: http://checkstyle.sourceforge.net/.
• 12: Cenqua. Cenqua Clover Code Coverage for Java, Available online at: http://www.cenqua.com/clover/.
• 13: Conradi, R., Li, J., Slyngstad, O.P.N., Kampenes, V.B., Bunse, C., Morisio, M. and Torchiano, M., Reflections on Conducting an International Survey of Software Engineering. In Proceedings of the 2005 International Symposium on Empirical Software Engineering (ISESE'05), (2005), 214--223.
• 14: John Daly , James Miller , Andrew Brooks , Marc Roper , Murray Wood, A Survey of Experiences amongst Object-Oriented Practitioners, Proceedings of the Second Asia Pacific Software Engineering Conference, p.137, December 06-09, 1995
• 15: Eytani, Y., Havelund, K., Stoller, S.D. and Ur, S. Toward a Framework and Benchmark for Testing Tools for Multi-Threaded Programs. To appear in Concurrency and Computation: Practice and Experience (2006).
• 16: Fink, A. The Survey Handbook. Sage Publications, 2003.
• 17: Foster, J.S., Position Paper: A Call for a Public Bug and Tool Registry. In 2005 SIGPLAN Workshop on the Evaluation of Software Defect Detection Tools, (2005), 70.
• 18: Fowler, F.J. Survey Research Methods. Sage Publications, 2002.
• 19: Geras, A.M., Smith, M.R. and Miller, J. A survey of software testing practices in Alberta. Canadian Journal of Electrical and Computer Engineering, 29, 3 (2004), 183--191.
• 20: Lindsay Groves , Ray Nickson , Greg Reeve , Steve Reeves , Mark Utting, A Survey of Software Development Practices in the New Zealand Software Industry, Proceedings of the 2000 Australian Software Engineering Conference, p.189, April 28-30, 2000
• 21: Havelund, K. and Pressburger, T. Model Checking Java Programs using Java PathFinder. International Journal of Software Tools for Technology Transfer (STTT), 2, 4 (2000), 366--381.
• 22: Barbara A. Kitchenham , Shari Lawrence Pfleeger, Principles of survey research part 2: designing a survey, ACM SIGSOFT Software Engineering Notes, v.27 n.1, p.18-20, January 2002 [doi>10.1145/566493.566495]
• 23: Doug Lea, The java.util.concurrent synchronizer framework, Science of Computer Programming, v.58 n.3, p.293-309, December 2005 [doi>10.1016/j.scico.2005.03.007]
• 24: Timothy C. Lethbridge , Susan Elliott Sim , Janice Singer, Studying Software Engineers: Data Collection Techniques for Software Field Studies, Empirical Software Engineering, v.10 n.3, p.311-341, July 2005 [doi>10.1007/s10664-005-1290-x]
• 25: Long, B., Strooper, P. and Wildman, L. A Method for Verifying Concurrent Java Components. To appear in Concurrency and Computation: Practice and Experience (2006).
• 26: Microsoft Research. Zing, Available online at: http://research.microsoft.com/zing/.
• 27: S. P. Ng , T. Murnane , K. Reed , D. Grant , T. Y. Chen, A Preliminary Survey on Software Testing Practices in Australia, Proceedings of the 2004 Australian Software Engineering Conference (ASWEC'04), p.116, April 13-16, 2004
• 28: OpenQA. Selenium, Available online at: http://www.openqa.org/selenium/.
• 29: Teade Punter , Marcus Ciolkowski , Bernd Freimut , Isabel John, Conducting On-line Surveys in Software Engineering, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.80, September 30-October 01, 2003
• 30: Quality Aussurance Institute, Q.A.I. Status of software testing, Available online at: http://www.qaiusa.com/bill-perry-downland/word%20download%20files/Test%20Survey.doc, 2002.
• 31: Roubtsov, V. EMMA: a free Java code coverage tool, Available online at: http://emma.sourceforge.net/.
• 32: SAT Research Group at Princeton University. ZChaff, Available online at: http://www.princeton.edu/~chaff/zchaff.html.
• 33: Software Testing Institue, S.T.I. 2002 Software Testing Institute survey, Available online at: http://www.softwaretestinginstitute.com/2002IndustrySurveyResults/STI2002IndustrySurveyResults_frame.htm, 2002.
• 34: Ian Sommerville, Software Engineering (7th Edition), Pearson Addison Wesley, 2004
• 35: Spacco, J., Hovemeyer, D. and Pugh, W., Bug Specimens are Important. In 2005 SIGPLAN Workshop on the Evaluation of Software Defect Detection Tools, (2005), 71--72.
• 36: Stonybrook State University of New York. Concurrency Factory, Available online at: http://www.cs.sunysb.edu/~concurr/.
• 37: Richard Torkar , Stefan Mankefors, A Survey on Testing and Reuse, Proceedings of the IEEE International Conference on Software-Science, Technology & Engineering, p.164, November 04-05, 2003
• 38: Ur, S. and Eytani, Y., Compiling a Benchmark of Documented Multi-threaded Bugs. In Proceedings of the 2nd International Workshop on Parallel and Distributed Systems: Testing and Debugging (PADTAD), (2004).
• 39: Sira Vegas , Victor Basili, A Characterisation Schema for Software Testing Techniques, Empirical Software Engineering, v.10 n.4, p.437-466, October 2005 [doi>10.1007/s10664-005-3862-1]
• 40: Vegas, S., Juristo, N. and Basili, V. A Process for Identifying Relevant Information for a Repository: A Case Study for Testing Techniques. In Aurum, A., Jeffery, R., Wohlin, C. and Handzic, M. eds. Managing Software Engineering Knowledge, Springer-Verlag, 2003, 199--230.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Validation

General Terms:
Experimentation, Verification

Keywords:
concurrency, survey, validation technologies, verification