Evaluating static analysis defect warnings on production software

ABSTRACT

Static analysis tools for software defect detection are becoming widely used in practice. However, there is little public information regarding the experimental evaluation of the accuracy and value of the warnings these tools report. In this paper, we discuss the warnings found by FindBugs, a static analysis tool that finds defects in Java programs. We discuss the kinds of warnings generated and the classification of warnings into false positives, trivial bugs and serious bugs. We also provide some insight into why static analysis tools often detect true but trivial bugs, and some information about defect warnings across the development lifetime of software release. We report data on the defect warnings in Sun's Java 6 JRE, in Sun's Glassfish JEE server, and in portions of Google's Java codebase. Finally, we report on some experiences from incorporating static analysis into the software development process at Google.

REFERENCES

• 1: A. Almossawi, K. Lim, and T. Sinha. Analysis tool evaluation: Coverity prevent, May 2006. http://www.cs.cmu.edu/ aldrich/courses/654/tools/cure-coverity-06.pdf.
• 2: K. I. Boudnik. Static analyzers comparison, October 2006. http://weblogs.java.net/blog/cos/archive/2006/10/static analyzer.html.
• 3: Benjamin Chelf , Dawson Engler , Seth Hallem, How to write system-specific, static checkers in metal, Proceedings of the 2002 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.51-60, November 18-19, 2002, Charleston, South Carolina, USA
• 4: T. Copeland. PMD Applied. Centennial Books, November 2005.
• 5: Dawson Engler , David Yu Chen , Seth Hallem , Andy Chou , Benjamin Chelf, Bugs as deviant behavior: a general approach to inferring errors in systems code, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 6: Isil Dillig , Thomas Dillig , Alex Aiken, Static error detection using semantic inconsistency inference, Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation, June 10-13, 2007, San Diego, California, USA
• 7: Zhenmin Li , Lin Tan , Xuanhui Wang , Shan Lu , Yuanyuan Zhou , Chengxiang Zhai, Have things changed now?: an empirical study of bug characteristics in modern open source software, Proceedings of the 1st workshop on Architectural and system support for improving software dependability, p.25-33, October 21-21, 2006, San Jose, California [doi>10.1145/1181309.1181314]
• 8: R. O'Callahan. Static analysis and scary headlines, September 2006. http://weblogs.mozillazine.org/roc/archives/2006/09/static analysis and scary head.html.
• 9: Nick Rutar , Christian B. Almazan , Jeffrey S. Foster, A Comparison of Bug Finding Tools for Java, Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE'04), p.245-256, November 02-05, 2004 [doi>10.1109/ISSRE.2004.1]
• 10: Jaime Spacco , David Hovemeyer , William Pugh, Tracking defect warnings across versions, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138014]
• 11: S. Wagner, F. Deissenboeck, M. A. J. Wimmer, and M. Schwalb. An evaluation of bug pattern tools for java, January 2007. unpublished.
• 12: S. Wagner, J. Jurjens, C. Koller, and P. Trischberger. Comparing bug finding tools with reviews and tests. In Proc. 17th International Conference on Testing of Communicating Systems, pages 40--55, 2005.
• 13: Misha Zitser , Richard Lippmann , Tim Leek, Testing static analysis tools using exploitable buffer overflows from open source code, Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, October 31-November 06, 2004, Newport Beach, CA, USA

INDEX TERMS

Primary Classification:
  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.2 Semantics of Programming Languages
          Subjects: Program analysis

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

General Terms:
Experimentation, Reliability, Security

Keywords:
FindBugs, Java, bug patterns, bugs, false positives, software defects, software quality, static analysis