Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Software engineering researchers have long been interested in where and why bugs occur in code, and in predicting where they might turn up next. Historical bug-occurence data has been key to this research. Bug tracking systems, and code version histories, record when, how and by whom bugs were fixed; from these sources, datasets that relate file changes to bug fixes can be extracted. These historical datasets can be used to test hypotheses concerning processes of bug introduction, and also to build statistical bug prediction models. Unfortunately, processes and humans are imperfect, and only a fraction of bug fixes are actually labelled in source code version histories, and thus become available for study in the extracted datasets. The question naturally arises, are the bug fixes recorded in these historical datasets a fair representation of the full population of bug fixes? In this paper, we investigate historical data from several software projects, and find strong evidence of systematic bias. We then investigate the potential effects of "unfair, imbalanced" datasets on the performance of prediction techniques. We draw the lesson that bias is a critical problem that threatens both the effectiveness of processes that rely on biased datasets to build prediction models and the generalizability of hypotheses tested on biased data.

REFERENCES

• 1: A. Agresti and B. Coull. Approximate Is Better Than "Exact" for Interval Estimation ofBinomial Proportions. The American Statistician, 52(2), 1998.
• 2: C. Ambroise and G. McLachlan. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences,99(10):6562--6566, 2002.
• 3: A. Bachmann and A. Bernstein. Data retrieval, processing and linking for software process dataanalysis. Technical report, University of Zurich, 2009. Published May, 2009. http://www.ifi.uzh.ch/ddis/people/adrian-bachmann/pdq/.
• 4: Adrian Bachmann , Abraham Bernstein, Software process data quality and characteristics: a historical view on open and closed source projects, Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops, August 24-25, 2009, Amsterdam, The Netherlands [doi>10.1145/1595808.1595830]
• 5: V. Basili, G. Caldiera, and H. Rombach. The Goal Question Metric Approach. Encyclopedia of Software Engineering, 1:528--532, 1994.
• 6: V. Basili and R. Selby Jr. Data collection and analysis in software research and management. Proc. of the American Statistical Association and BiomeasureSociety Joint Statistical Meetings, 1984.
• 7: V. Basili and D. Weiss. A methodology for collecting valid software engineering data. IEEE Transactions on Software Engineering, 10(6):728--738,1984.
• 8: Y. Benjamini and Y. Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 1995.
• 9: R. Berk. An introduction to sample selection bias in sociological data. American Sociological Review, 48(3):386--398, 1983.
• 10: Bugzilla Fields, http://www.eclipse.org/tptp/home/documents/process/development/bugzilla.html.
• 11: C. Catal and B. Diri. A systematic review of software fault prediction studies. Expert Systems With Applications, 2008.
• 12: W. J. Conover. Practical Nonparametric Statistics. John Wiley&Sons, 1971.
• 13: Davor Cubranic , Gail C. Murphy , Janice Singer , Kellogg S. Booth, Hipikat: A Project Memory for Software Development, IEEE Transactions on Software Engineering, v.31 n.6, p.446-465, June 2005 [doi>10.1109/TSE.2005.71]
• 14: Valentin Dallmeier , Thomas Zimmermann, Extraction of bug localization benchmarks from history, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA [doi>10.1145/1321631.1321702]
• 15: S. Dowdy, S. Wearden, and D. Chilko. Statistics for research. John Wiley&Sons, third edition, 2004.
• 16: Marc Eaddy , Thomas Zimmermann , Kaitlin D. Sherwood , Vibhav Garg , Gail C. Murphy , Nachiappan Nagappan , Alfred V. Aho, Do Crosscutting Concerns Cause Defects?, IEEE Transactions on Software Engineering, v.34 n.4, p.497-515, July 2008 [doi>10.1109/TSE.2008.36]
• 17: P. Easterbrook, J. Berlin, R. Gopalan, and D. Matthews. Publication bias in clinical research. Lancet, 337(8746):867--72, 1991.
• 18: S. Easterbrook, J. Singer, M. Storey, and D. Damian. Selecting Empirical Methods for Software Engineering Research. Guide to Advanced Empirical Software Engineering, 2007.
• 19: Michael Fischer , Martin Pinzger , Harald Gall, Populating a Release History Database from Version Control and Bug Tracking Systems, Proceedings of the International Conference on Software Maintenance, p.23, September 22-26, 2003
• 20: L. Gasser and G. Ripoche. Distributed collective practices and free/open-source software problem management: perspectives and methods. Proceedings of the Conference on Coopration, Innovations et Technologies, 2003.
• 21: M. Grabe, S. Zhou, and B. Barnett. Explicating sensationalism in television news: Content and the bells and whistles of form. Journal of Broadcasting&Electronic Media, 45:635, 2001.
• 22: Robert B. Grady , Deborah L. Caswell, Software metrics: establishing a company-wide program, Prentice-Hall, Inc., Upper Saddle River, NJ, 1987
• 23: J. Heckman. Sample Selection Bias as a Specification Error. Econometrica, 47(1):153--161, 1979.
• 24: Israel Herraiz , Gregorio Robles , Jesus M. Gonzalez-Barahona, Towards predictor models for large libre software projects, ACM SIGSOFT Software Engineering Notes, v.30 n.4, July 2005 [doi>10.1145/1082983.1083168]
• 25: Sunghun Kim , Kai Pan , E. E. James Whitehead, Jr., Memories of bug fixes, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181781]
• 26: Sunghun Kim , Thomas Zimmermann , Kai Pan , E. James Jr. Whitehead, Automatic Identification of Bug-Introducing Changes, Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering, p.81-90, September 18-22, 2006 [doi>10.1109/ASE.2006.23]
• 27: Sunghun Kim , Thomas Zimmermann , E. James Whitehead Jr. , Andreas Zeller, Predicting Faults from Cached History, Proceedings of the 29th international conference on Software Engineering, p.489-498, May 20-26, 2007 [doi>10.1109/ICSE.2007.66]
• 28: A. Günes Koru , Hongfang Liu, An investigation of the effect of module size on defect prediction using static measures, ACM SIGSOFT Software Engineering Notes, v.30 n.4, July 2005 [doi>10.1145/1082983.1083172]
• 29: A. Gunes Koru , Jeff Tian, Defect Handling in Medium and Large Open Source Projects, IEEE Software, v.21 n.4, p.54-61, July 2004 [doi>10.1109/MS.2004.12]
• 30: M. R. Levy. The Methodology and Performance of Election Day Polls. Public Opinion Quarterly, 47(1):54--67, 1983.
• 31: Gernot A. Liebchen , Martin Shepperd, Data sets and data quality in software engineering, Proceedings of the 4th international workshop on Predictor models in software engineering, May 12-13, 2008, Leipzig, Germany [doi>10.1145/1370788.1370799]
• 32: Gernot Liebchen , Bheki Twala , Martin Shepperd , Michelle Cartwright , Mark Stephens, Filtering, Robust Filtering, Polishing: Techniques for Addressing Quality in Software Data, Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, p.99-106, September 20-21, 2007 [doi>10.1109/ESEM.2007.48]
• 33: R. Little and D. Rubin. Statistical analysis with missing data. Technometrics, 45(4):364--365, 2003.
• 34: Thomas M. Mitchell, Machine Learning, McGraw-Hill, Inc., New York, NY, 1997
• 35: A. Mockus. Missing Data in Software Engineering. Empirical Methods in Software Engineering. The MIT Press), 2000.
• 36: Raimund Moser , Witold Pedrycz , Giancarlo Succi, A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction, Proceedings of the 30th international conference on Software engineering, May 10-18, 2008, Leipzig, Germany [doi>10.1145/1368088.1368114]
• 37: Stephan Neuhaus , Thomas Zimmermann , Christian Holler , Andreas Zeller, Predicting vulnerable software components, Proceedings of the 14th ACM conference on Computer and communications security, October 28-31, 2007, Alexandria, Virginia, USA [doi>10.1145/1315245.1315311]
• 38: M. Nick and C. Tautz. Practical evaluation of an organizational memory using the goal-question-metric technique. Lecture notes in computer science, pages 138--147, 1999.
• 39: R. Nickerson. Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2:175--220, 1998.
• 40: Dewayne E. Perry , Adam A. Porter , Lawrence G. Votta, Empirical studies of software engineering: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.345-355, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336586]
• 41: Promise '08: Proceedings of the 4th international workshop on predictor models in software engineering, 2008. Eds. B. Boetticher and T. Ostrand.
• 42: Promise Dataset, http://promisedata.org.
• 43: Adrian Schröter , Thomas Zimmermann , Andreas Zeller, Predicting component failures at design time, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159739]
• 44: Forrest Shull , Janice Singer , Dag I.K. Sjøberg, Guide to Advanced Empirical Software Engineering, Springer-Verlag New York, Inc., Secaucus, NJ, 2007
• 45: R. A. Singleton, Jr. and B. C. Straits. Approaches to Social Research. Oxford University Press, 2005.
• 46: Jacek Śliwerski , Thomas Zimmermann , Andreas Zeller, When do changes induce fixes?, Proceedings of the 2005 international workshop on Mining software repositories, p.1-5, May 17-17, 2005, St. Louis, Missouri [doi>10.1145/1083142.1083147]
• 47: K. Weiss. Confounding, ascertainment bias, and the blind quest for a genetic 'fountain of youth'. Annals of Medicine, 35:532--544, 2003.
• 48: Bianca Zadrozny, Learning and evaluating classifiers under sample selection bias, Proceedings of the twenty-first international conference on Machine learning, p.114, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015425]
• 49: Thomas Zimmermann , Rahul Premraj , Andreas Zeller, Predicting Defects for Eclipse, Proceedings of the Third International Workshop on Predictor Models in Software Engineering, p.9, May 20-26, 2007 [doi>10.1109/PROMISE.2007.10]
• 50: T. Zimmermann and P. Weißgerber. Preprocessing CVS data for fine-grained analysis. In Proceedings of the International Workshop on Mining Software Repositories, 2004.
• 51: Shoshana Zuboff, In the age of the smart machine: the future of work and power, Basic Books, Inc., New York, NY, 1988

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Process metrics

General Terms:
Experimentation, Measurement

Keywords:
bias