Analysis of the influence of communication between researchers on experiment replication

ABSTRACT

The replication of experiments is a key undertaking in SE. Successful replications enable a discipline's body of knowledge to grow, as the results are added to those of earlier replications. However, replication is extremely difficult in SE, primarily because it is difficult to get a setting that is exactly the same as in the original experiment. Consequently, changes have to be made to the experiment to adapt it to the new site. To be able to replicate an experiment, information also has to be transmitted (usually orally and in writing) between the researchers who ran the experiment earlier and the ones who are going to replicate the experiment. This article examines the influence of the type of communication there is between experimenters on how successful a replication is. We have studied three replications of the same experiment in which different types of communication were used.

REFERENCES

• 1: V.R. Basili, R.W. Selby. Comparing the Effectiveness of Software Testing Strategies. Department of Computer Science. University of Maryland. Technical Report TR-1501. College Park. May 1985.
• 2: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 3: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 4: M. Ciolkowski, C. Differding, O. Laitenberger, J. Muench. Empirical Investigation of Perspective-Based Reading: A Replicated Experiment. ISERN Technical Report. ISERN-97-13. 1997.
• 5: R. Conradi, V.R. Basili, J. Carver, F. Shull, G.H. Travassos. A Pragmatic Documents Standard for an Experience Library: Roles, Documents, Contents and Structure. University of Maryland Technical Report. CS-TR-4235. 2001.
• 6: M. Jørgensen. A Review of Studies on Expert Estimation of Software Development Effort. Journal of Systems and Software. 70 (1-2): 37--60. 2004.
• 7: N. Juristo, A.M. Moreno. Basics of Software Engineering Experimentation. Kluwer. 2001.
• 8: Natalia Juristo , Ana M. Moreno , Sira Vegas, Reviewing 25 Years of Testing Technique Experiments, Empirical Software Engineering, v.9 n.1-2, p.7-44, March 2004 [doi>10.1023/B:EMSE.0000013513.48963.1b]
• 9: N. Juristo, S. Vegas. Functional testing, structural testing and code reading: What fault type do they each detect? In Empirical Methods and Studies in Software Engineering- Experiences from ESERNET.. R. Conradi and A.I. Wang Eds. LNCS 2785. Pp 235--261. Springer-Verlag. 2003.
• 10: E. Kamsties, C. Lott. An empirical evaluation of three defect detection techniques. Technical Report ISERN 95-02, Dept. Computer Science, University of Kaiserslautern, May 1995.
• 11: Erik Kamsties , Christopher M. Lott, An Empirical Evaluation of Three Defect-Detection Techniques, Proceedings of the 5th European Software Engineering Conference, p.362-383, September 25-28, 1995
• 12: Oliver Laitenberger , Dieter Rombach, (Quasi-)experimental studies in industrial settings, Lecture notes on empirical software engineering, World Scientific Publishing Co., Inc., River Edge, NJ, 2003
• 13: James Miller, Applying meta-analytical procedures to software engineering experiments, Journal of Systems and Software, v.54 n.1, p.29-39, Sept. 30, 2000 [doi>10.1016/S0164-1212(00)00024-8]
• 14: L.M. Pickard,, B.A. Kitchenham, P.W. Jones. Combining Empirical Results in Software Engineering. Information and Software Technology. 40 (14): 811--821. 1998.
• 15: Adam A. Porter , Philip M. Johnson, Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies, IEEE Transactions on Software Engineering, v.23 n.3, p.129-145, March 1997 [doi>10.1109/32.585501]
• 16: M. Roper, M. Wood, J. Miller. An empirical evaluation of defect detection techniques. Information and Software Technology. 39: 763--775. 1997.
• 17: Forrest Shull , Jeffrey Carver , Guilherme H. Travassos , José Carlos Maldonado , Reidar Conradi , Victor R. Basili, Replicated studies: building a body of knowledge about software reading techniques, Lecture notes on empirical software engineering, World Scientific Publishing Co., Inc., River Edge, NJ, 2003
• 18: Forrest Shull , Manoel G. Mendoncça , Victor Basili , Jeffrey Carver , José C. Maldonado , Sandra Fabbri , Guilherme Horta Travassos , Maria Cristina Ferreira, Knowledge-Sharing Issues in Experimental Software Engineering, Empirical Software Engineering, v.9 n.1-2, p.111-137, March 2004 [doi>10.1023/B:EMSE.0000013516.80487.33]
• 19: S. Vegas. Maduración de Conocimiento Mediante una Familia de Experimentos. Jornadas Iberoamericanas en Ingeniería del Software Ingeniería del Conocimiento (JIISIC'04). Pp 487--500. An English version is available at http://is.ls.fi.upm.es/udis/miembros/sira/cv.html
• 20: Murray Wood , Marc Roper , Andrew Brooks , James Miller, Comparing and combining software defect detection techniques: a replicated empirical study, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.262-277, September 22-25, 1997, Zurich, Switzerland

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.0 General

General Terms:
Experimentation

Keywords:
empirical studies, experimentation, replication