Predicting accurate and actionable static analysis warnings: an experimental approach

ABSTRACT

Static analysis tools report software defects that may or may not be detected by other verification methods. Two challenges complicating the adoption of these tools are spurious false positive warnings and legitimate warnings that are not acted on. This paper reports automated support to help address these challenges using logistic regression models that predict the foregoing types of warnings from signals in the warnings and implicated code. Because examining many potential signaling factors in large software development settings can be expensive, we use a screening methodology to quickly discard factors with low predictive power and cost-effectively build predictive models. Our empirical evaluation indicates that these models can achieve high accuracy in predicting accurate and actionable static analysis warnings, and suggests that the models are competitive with alternative models built without screening.

REFERENCES

• 1: FindBugs. http://findbugs.sourceforge.net/.
• 2: The R project for statistical computing. http://r-project.org/.
• 3: Nathaniel Ayewah , William Pugh , J. David Morgenthaler , John Penix , YuQian Zhou, Evaluating static analysis defect warnings on production software, Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.1-8, June 13-14, 2007, San Diego, California, USA [doi>10.1145/1251535.1251536]
• 4: Robert M. Bell , Thomas J. Ostrand , Elaine J. Weyuker, Looking for bugs in all the right places, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146246]
• 5: Dawson Engler , David Yu Chen , Seth Hallem , Andy Chou , Benjamin Chelf, Bugs as deviant behavior: a general approach to inferring errors in systems code, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 6: T. J. Hastie and D. Pregibon. Statistical Models in S. Wadsworth & Brooks/Cole, 1992.
• 7: Sarah Smith Heckman, Adaptively ranking alerts generated from automated static analysis, Crossroads, v.14 n.1, p.1-11, December 2007 [doi>10.1145/1349332.1349339]
• 8: D. W. Hosmer and S. Lemeshow. Applied Logistic Regression. John Wiley & Sons, 2nd ed., 2000.
• 9: David Hovemeyer , William Pugh, Finding bugs is easy, Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, October 24-28, 2004, Vancouver, BC, CANADA [doi>10.1145/1028664.1028717]
• 10: R. A. Johnson , D. W. Wichern, Applied multivariate statistical analysis, Prentice-Hall, Inc., Upper Saddle River, NJ, 1988
• 11: Sunghun Kim , Michael D. Ernst, Prioritizing Warning Categories by Analyzing Software History, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.27, May 20-26, 2007 [doi>10.1109/MSR.2007.26]
• 12: Sunghun Kim , Michael D. Ernst, Which warnings should I fix first?, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287633]
• 13: Ted Kremenek , Ken Ashcraft , Junfeng Yang , Dawson Engler, Correlation exploitation in error ranking, Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, October 31-November 06, 2004, Newport Beach, CA, USA
• 14: T. Kremenek and D. Engler. Z-Ranking: Using statistical analysis to counter the impact of static analysis approximations. In Proc. 10th Static Analysis Symp., 2003.
• 15: L. Z. Markosian, O. O'Malley, J. Penix, andW. Brew. Hosted services for advanced V&V technologies: An approach to achieving adoption without the woes of usage. In Proc. ICSE Workshop on Adoption-Centric Softw. Eng., 2003.
• 16: Nachiappan Nagappan , Thomas Ball , Andreas Zeller, Mining metrics to predict component failures, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134349]
• 17: J. Neter, M. H. Kutner, C. J. Nachtsheim, and W. Wasserman. Applied Linear Statistical Models. Irwin, 4th edition, 1996.
• 18: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Where the bugs are, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 19: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Automating algorithms for the identification of fault-prone files, Proceedings of the 2007 international symposium on Software testing and analysis, July 09-12, 2007, London, United Kingdom [doi>10.1145/1273463.1273493]
• 20: Joseph R. Ruthruff , Sebastian Elbaum , Gregg Rothermel, Experimental program analysis: a new program analysis paradigm, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146245]
• 21: Jaime Spacco , David Hovemeyer , William Pugh, Tracking defect warnings across versions, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138014]
• 22: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Statistical methods

  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.2 Semantics of Programming Languages
          Subjects: Program analysis

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

General Terms:
Experimentation, Reliability

Keywords:
experimental program analysis, logistic regression analysis, screening, software quality, static analysis tools