Evaluating guidelines for empirical software engineering studies

ABSTRACT

Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve.Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed.Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based on a set of questions derived by brainstorming. The inspection using the Author perspective reviewed each section of the guidelines sequentially. Results. The question-based perspective inspections detected 42 issues where the guidelines would benefit from amendment or clarification and 8 defects.Conclusions. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. Software engineering researchers need to be cautious about adopting reporting guidelines that differ from those used by other disciplines. The current guidelines need to be revised and the revised guidelines need to be subjected to further theoretical and empirical validation. Perspective-based inspection is a useful validation method but the practitioner/consultant perspective presents difficulties.

REFERENCES

• 1: Zeiad Abdelnabi , Giovanni Cantone , Marcus Ciolkowski , Dieter Rombach, Comparing Code Reading Techniques Applied to Object-Oriented Software Frameworks with Regard to Effectiveness and Defect Detection Rate, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.239-248, August 19-20, 2004 [doi>10.1109/ISESE.2004.15]
• 2: Silvia Abrahao , Geert Poels , Oscar Pastor, Assessing the Reproducibility and Accuracy of Functional Size Measurement Methods through Experimentation, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.189-198, August 19-20, 2004 [doi>10.1109/ISESE.2004.12]
• 3: Tore Dybå, Vigdis By Kampenes, Dag I.K. Sjøberg. A systematic review of statistical power in software engineering experiments. Information and Software Technology, in press.
• 4: Peter Harris. Designing and Reporting Experiments in Psychology. 2nd Edition, Open University Press, 2002.
• 5: James Hartley. Current findings from research on structured abstracts. J. Med. Libr. Assoc. 92(3), July 2004, pp 368--371.
• 6: Andreas Jedlitschka and Dietmar Pfahl. Reporting Guidelines for Controlled Experiments in Software Engineering. IESEReport IESE-035.5/E, 2005.
• 7: Barbara A. Kitchenham , Shari Lawrence Pfleeger , Lesley M. Pickard , Peter W. Jones , David C. Hoaglin , Khaled El Emam , Jarrett Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, v.28 n.8, p.721-734, August 2002 [doi>10.1109/TSE.2002.1027796]
• 8: David Moher, Kenneth F Schultz, Douglas Altman. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. The LANCET, 357, April 14, 2001, pp. 1191--1194.
• 9: Pickard, L.M., Kitchenham, B.A., and Jones, P. (1998) Combining Empirical Results in Software Engineering. Information and Software Technology. Vol 40 No 14, pp 811--821.
• 10: Patrick J. Schroeder , Pankaj Bolaki , Vijayram Gopu, Comparing the Fault Detection Effectiveness of N-way and Random Test Suites, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.49-59, August 19-20, 2004 [doi>10.1109/ISESE.2004.16]
• 11: Dag I. K. Sjoberg , Jo E. Hannay , Ove Hansen , Vigdis By Kampenes , Amela Karahasanovic , Nils-Kristian Liborg , Anette C. Rekdal, A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, v.31 n.9, p.733-753, September 2005 [doi>10.1109/TSE.2005.97]
• 12: Jan Verelst, The Influence of the Level of Abstraction on the Evolvability of Conceptual Models of Information Systems, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.17-26, August 19-20, 2004 [doi>10.1109/ISESE.2004.29]
• 13: Claes Wohlin , Håkan Petersson , Aybüke Aurum, Combining data from reading experiments in software inspections: a feasibility study, Lecture notes on empirical software engineering, World Scientific Publishing Co., Inc., River Edge, NJ, 2003
• 14: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software process

General Terms:
Experimentation, Management

Keywords:
controlled experiments, guidelines, perspective-based inspection, software engineering