On the success of empirical studies in the international conference on software engineering

ABSTRACT

Critiques of the quantity and quality of empirical evaluations in software engineering have existed for quite some time. However such critiques are typically not empirically evaluated. This paper fills this gap by empirically analyzing papers published by ICSE, the prime research conference on Software Engineering. We present quantitative and qualitative results of a quasi-random experiment of empirical evaluations over the lifetime of the conference. Our quantitative results show the quantity of empirical evaluation has increased over 29 ICSE proceedings but we still have room to improve the soundness of empirical evaluations in ICSE proceedings. Our qualitative results point to specific areas of improvement in empirical evaluations.

REFERENCES

• 1: V R Basili , R W Selby , D H Hutchens, Experimentation in software engineering, IEEE Transactions on Software Engineering, v.12 n.7, p.733-743, July 1986
• 2: Victor R. Basili, The Experimental Paradigm in Software Engineering, Proceedings of the International Workshop on Experimental Software Engineering Issues: Critical Assessment and Future Directions, p.3-12, September 14-18, 1992
• 3: Victor R. Basili, The role of experimentation in software engineering: past, current, and future, Proceedings of the 18th international conference on Software engineering, p.442-449, March 25-29, 1996, Berlin, Germany
• 4: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 5: Victor R. Basili , Forrest Shull , Filippo Lanubile, Using Experiments to Build a Body of Knowledge, Proceedings of the Third International Andrei Ershov Memorial Conference on Perspectives of System Informatics, p.265-282, July 06-09, 1999
• 6: L. Briand , E. Arisholm , S. Counsell , F. Houdek , P. Thévenod--fosse, Empirical Studies of Object-Oriented Artifacts, Methods,and Processes: State of the Art and Future Directions, Empirical Software Engineering, v.4 n.4, p.387-404, December 1999 [doi>10.1023/A:1009825923070]
• 7: Norman Fenton , Shari Lawrence Pfleeger, Software metrics (2nd ed.): a rigorous and practical approach, PWS Publishing Co., Boston, MA, 1997
• 8: Norman Fenton , Shari Lawrence Pfleeger , Robert L. Glass, Science and Substance: A Challenge to Software Engineers, IEEE Software, v.11 n.4, p.86-95, July 1994 [doi>10.1109/52.300094]
• 9: Glass G.V; et al; Meta-analysis in Social Research; Beverly Hills, CA; Sage, 1981.
• 10: Glass R.L et al; "Research in software engineering: an Analysis of the Literature" IST 44, pp491--506, 2002.
• 11: Ross Jeffery , Louise Scott, Has Twenty-five Years of Empirical Software Engineering Made a Difference?, Proceedings of the Ninth Asia-Pacific Software Engineering Conference, p.539, December 04-06, 2002
• 12: Juristo N, et al; Basics of Software Engineering Experimentation; Kluwer Acad. Pub. Boston MA, 2001.
• 13: Barbara A. Kitchenham , Shari Lawrence Pfleeger , Lesley M. Pickard , Peter W. Jones , David C. Hoaglin , Khaled El Emam , Jarrett Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, v.28 n.8, p.721-734, August 2002 [doi>10.1109/TSE.2002.1027796]
• 14: Walter F. Tichy , Paul Lukowicz , Lutz Prechelt , Ernst A. Heinz, Experimental evaluation in computer science: a quantitative study, Journal of Systems and Software, v.28 n.1, p.9-18, Jan. 1995 [doi>10.1016/0164-1212(94)00111-Y]
• 15: Milton et al. Introduction to Statistics; McGraw-Hill, 1997.
• 16: Patton M.Q; Qualitative Research & Evaluation Methods 3rd Ed.; Sage Publications, California, 2002.
• 17: Dewayne E. Perry , Susan Elliott Sim , Steve M. Easterbrook, Case Studies for Software Engineers, Proceedings of the 26th International Conference on Software Engineering, p.736-738, May 23-28, 2004
• 18: Dewayne E. Perry , Adam A. Porter , Lawrence G. Votta, Empirical studies of software engineering: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.345-355, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336586]
• 19: Shari Lawrence Pfleeger, Soup or Art? The Role of Evidential Force in Empirical Software Engineering, IEEE Software, v.22 n.1, p.66-73, January 2005 [doi>10.1109/MS.2005.19]
• 20: Judith Segal , Antony Grinyer , Helen Sharp, The type of evidence produced by empirical software engineers, Proceedings of the 2005 workshop on Realising evidence-based software engineering, p.1-4, May 17-17, 2005, St. Louis, Missouri
• 21: Mary Shaw, Writing good software engineering research papers: minitutorial, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 22: Dag I. K. Sjoberg , Jo E. Hannay , Ove Hansen , Vigdis By Kampenes , Amela Karahasanovic , Nils-Kristian Liborg , Anette C. Rekdal, A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, v.31 n.9, p.733-753, September 2005 [doi>10.1109/TSE.2005.97]
• 23: Walter F. Tichy, Should Computer Scientists Experiment More?, Computer, v.31 n.5, p.32-40, May 1998 [doi>10.1109/2.675631]
• 24: Robert J. Walker , Lionel C. Briand , David Notkin , Carolyn B. Seaman , Walter F. Tichy, Panel: empirical validation: what, why, when, and how, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 25: Yancey, J. M; "Ten Rules of Reading Clinical Research Reports"; American J. Orthodontics and Dentofacial Orthopedics, V.109, No.5: pp 558--564, 1996.
• 26: Yin R.K; Case Study Research: Design and Methods, 3/e Thousand Oaks, CA: Sage Publications, 2002.
• 27: Marvin V. Zelkowitz , Dolores R. Wallace , David W. Binkley, Experimental validation of new software technology, Lecture notes on empirical software engineering, World Scientific Publishing Co., Inc., River Edge, NJ, 2003

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.0 General

General Terms:
Experimentation, Measurement

Keywords:
empirical evaluation