Which warnings should I fix first?

ABSTRACT

Automatic bug-finding tools have a high false positive rate: most warnings do not indicate real bugs. Usually bug-finding tools assign important warnings high priority. However, the prioritization of tools tends to be ineffective. We observed the warnings output by three bug-finding tools, FindBugs, JLint, and PMD, for three subject programs, Columba, Lucene, and Scarab. Only 6%, 9%, and 9% of warnings are removed by bug fix changes during 1 to 4 years of the software development. About 90% of warnings remain in the program or are removed during non-fix changes -- likely false positive warnings. The tools' warning prioritization is little help in focusing on important warnings: the maximum possible precision by selecting high-priority warning instances is only 3%, 12%, and 8% respectively.

In this paper, we propose a history-based warning prioritization algorithm by mining warning fix experience that is recorded in the software change history. The underlying intuition is that if warnings from a category are eliminated by fix-changes, the warnings are important. Our prioritization algorithm improves warning precision to 17%, 25%, and 67% respectively.

REFERENCES

• 1: Ethem Alpaydin, Introduction to Machine Learning (Adaptive Computation and Machine Learning), The MIT Press, 2004
• 2: C. Artho, "Jlint -- Find Bugs in Java Programs," 2006, http://jlint.sourceforge.net/.
• 3: Jennifer Bevan , E. James Whitehead, Jr. , Sunghun Kim , Michael Godfrey, Facilitating software evolution research with kenyon, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 4: Cathal Boogerd , Leon Moonen, Prioritizing Software Inspection Results using Static Profiling, Proceedings of the Sixth IEEE International Workshop on Source Code Analysis and Manipulation, p.149-160, September 27-29, 2006 [doi>10.1109/SCAM.2006.22]
• 5: Kai Chen , Stephen R. Schach , Liguo Yu , Jeff Offutt , Gillian Z. Heller, Open-Source Change Logs, Empirical Software Engineering, v.9 n.3, p.197-210, September 2004 [doi>10.1023/B:EMSE.0000027779.70556.d0]
• 6: T. Copeland, PMD Applied: Centennial Books, 2005.
• 7: Davor Čubranić , Gail C. Murphy, Hipikat: recommending pertinent software development artifacts, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 8: Davor Cubranic , Gail C. Murphy , Janice Singer , Kellogg S. Booth, Hipikat: A Project Memory for Software Development, IEEE Transactions on Software Engineering, v.31 n.6, p.446-465, June 2005 [doi>10.1109/TSE.2005.71]
• 9: Michael Fischer , Martin Pinzger , Harald Gall, Populating a Release History Database from Version Control and Bug Tracking Systems, Proceedings of the International Conference on Software Maintenance, p.23, September 22-26, 2003
• 10: Cormac Flanagan , K. Rustan M. Leino , Mark Lillibridge , Greg Nelson , James B. Saxe , Raymie Stata, Extended static checking for Java, Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation, June 17-19, 2002, Berlin, Germany
• 11: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 12: David Hovemeyer , William Pugh, Finding bugs is easy, Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, October 24-28, 2004, Vancouver, BC, CANADA [doi>10.1145/1028664.1028717]
• 13: Sunghun Kim , Michael D. Ernst, Prioritizing Warning Categories by Analyzing Software History, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.27, May 20-26, 2007 [doi>10.1109/MSR.2007.26]
• 14: Sunghun Kim , Kai Pan , E. E. James Whitehead, Jr., Memories of bug fixes, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181781]
• 15: Sunghun Kim , Thomas Zimmermann , Kai Pan , E. James Jr. Whitehead, Automatic Identification of Bug-Introducing Changes, Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering, p.81-90, September 18-22, 2006 [doi>10.1109/ASE.2006.23]
• 16: Sunghun Kim , Thomas Zimmermann , E. James Whitehead Jr. , Andreas Zeller, Predicting Faults from Cached History, Proceedings of the 29th international conference on Software Engineering, p.489-498, May 20-26, 2007 [doi>10.1109/ICSE.2007.66]
• 17: T. Kremenek and D. R. Engler, "Z-ranking: Using statistical analysis to counter the impact of static analysis approximations," Proc. of the 10th International Symposium on Static Analysis (SAS 2003), San Diego, CA, USA, pp. 295--315, 2003.
• 18: Zhenmin Li , Lin Tan , Xuanhui Wang , Shan Lu , Yuanyuan Zhou , Chengxiang Zhai, Have things changed now?: an empirical study of bug characteristics in modern open source software, Proceedings of the 1st workshop on Architectural and system support for improving software dependability, p.25-33, October 21-21, 2006, San Jose, California [doi>10.1145/1181309.1181314]
• 19: Audris Mockus , Lawrence G. Votta, Identifying Reasons for Software Changes Using Historic Databases, Proceedings of the International Conference on Software Maintenance (ICSM'00), p.120, October 11-14, 2000
• 20: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Where the bugs are, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 21: Nick Rutar , Christian B. Almazan , Jeffrey S. Foster, A Comparison of Bug Finding Tools for Java, Proceedings of the 15th International Symposium on Software Reliability Engineering, p.245-256, November 02-05, 2004 [doi>10.1109/ISSRE.2004.1]
• 22: Jacek Śliwerski , Thomas Zimmermann , Andreas Zeller, When do changes induce fixes?, Proceedings of the 2005 international workshop on Mining software repositories, p.1-5, May 17-17, 2005, St. Louis, Missouri
• 23: Jaime Spacco , David Hovemeyer , William Pugh, Tracking defect warnings across versions, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138014]
• 24: Chadd C. Williams , Jeffrey K. Hollingsworth, Automatic Mining of Source Code Repositories to Improve Bug Finding Techniques, IEEE Transactions on Software Engineering, v.31 n.6, p.466-480, June 2005 [doi>10.1109/TSE.2005.63]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Restructuring, reverse engineering, and reengineering

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
bug, bug-finding tool, fault, fix, patterns, prediction