On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques

Full text

Pdf (397 KB)

Source

ESEM archive
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement table of contents

Kaiserslautern, Germany

SESSION: Testing and analysis table of contents

Pages 41-50

Year of Publication: 2008

ISBN:978-1-59593-971-5

Authors

Sarah Heckman	North Carolina State University, Raleigh, NC, USA
Laurie Williams	North Carolina State University, Raleigh, NC, USA

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
ACM: Association for Computing Machinery

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 12, Downloads (12 Months): 100, Citation Count: 0

Additional Information:

abstract references index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1414004.1414013 What is a DOI?

ABSTRACT

Benchmarks provide an experimental basis for evaluating software engineering processes or techniques in an objective and repeatable manner. We present the FAULTBENCH v0.1 benchmark, as a contribution to current benchmark materials, for evaluation and comparison of techniques that prioritize and classify alerts generated by static analysis tools. Static analysis tools may generate an overwhelming number of alerts, the majority of which are likely to be false positives (FP). Two FP mitigation techniques, alert prioritization and classification, provide an ordering or classification of alerts, identifying those likely to be anomalies. We evaluate FAULTBENCH using three versions of a FP mitigation technique within the AWARE adaptive prioritization model. Individual FAULTBENCH subjects vary in their optimal FP mitigation techniques. Together, FAULTBENCH subjects provide a precise and general evaluation of FP mitigation techniques.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Nathaniel Ayewah , William Pugh , J. David Morgenthaler , John Penix , YuQian Zhou, Evaluating static analysis defect warnings on production software, Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.1-8, June 13-14, 2007, San Diego, California, USA [doi>10.1145/1251535.1251536]
	2	Boehm , Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2003
	3	G. Boetticher, T. Menzies, and T. Ostrand, "PROMISE Repository of Empirical Software Engineering Data," http://promisedata.org/ repository, West Virginia University, Department of Computer Science, 2007.
	4	Cathal Boogerd , Leon Moonen, Prioritizing Software Inspection Results using Static Profiling, Proceedings of the Sixth IEEE International Workshop on Source Code Analysis and Manipulation, p.149-160, September 27-29, 2006 [doi>10.1109/SCAM.2006.22]
	5	Sarah Smith Heckman, Adaptively ranking alerts generated from automated static analysis, Crossroads, v.14 n.1, p.1-11, December 2007 [doi>10.1145/1349332.1349339]
	6	David Hovemeyer , William Pugh, Finding bugs is easy, Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications, October 24-28, 2004, Vancouver, BC, CANADA [doi>10.1145/1028664.1028717]
	7	Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
	8	IEEE, "IEEE Standard 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology," 1990.
	9	IEEE, "IEEE 1028-1997 (R2002) IEEE Standard for Software Reviews," 2002.
	10	Sunghun Kim , Michael D. Ernst, Prioritizing Warning Categories by Analyzing Software History, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.27, May 20-26, 2007 [doi>10.1109/MSR.2007.26]
	11	Sunghun Kim , Michael D. Ernst, Which warnings should I fix first?, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287633]
	12	Sunghun Kim , Thomas Zimmermann , E. James Whitehead Jr. , Andreas Zeller, Predicting Faults from Cached History, Proceedings of the 29th international conference on Software Engineering, p.489-498, May 20-26, 2007 [doi>10.1109/ICSE.2007.66]
	13	Ted Kremenek , Ken Ashcraft , Junfeng Yang , Dawson Engler, Correlation exploitation in error ranking, Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering, October 31-November 06, 2004, Newport Beach, CA, USA
	14	T. Kremenek and D. Engler, "Z-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations," Proceedings of the 10th International Static Analysis Symposium, San Diego, California, 2002.
	15	S. Lu, Z. Li, F. Oin, L. Tan, P. Zhou, and Y. Zhou, "BugBench: Benchmarks for Evaluating Bug Detection Tools," Proceedings of the Workshop on the Evaluation of Software Defect Detection Tools, Chicago, Illinois, 2005.
	16	Mayur Naik , Alex Aiken , John Whaley, Effective static race detection for Java, Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation, June 11-14, 2006, Ottawa, Ontario, Canada
	17	Gregg Rothermel , Roland J. Untch , Chengyun Chu, Prioritizing Test Cases For Regression Testing, IEEE Transactions on Software Engineering, v.27 n.10, p.929-948, October 2001 [doi>10.1109/32.962562]
	18	Susan Elliott Sim , Steve Easterbrook , Richard C. Holt, Using benchmarking to advance research: a challenge to software engineering, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
	19	Walter F. Tichy, Should Computer Scientists Experiment More?, Computer, v.31 n.5, p.32-40, May 1998 [doi>10.1109/2.675631]
	20	Stefan Wagner , Florian Deissenboeck , Michael Aichner , Johann Wimmer , Markus Schwalb, An Evaluation of Two Bug Pattern Tools for Java, Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation, p.248-257, April 09-11, 2008 [doi>10.1109/ICST.2008.63]
	21	Chadd C. Williams , Jeffrey K. Hollingsworth, Automatic Mining of Source Code Repositories to Improve Bug Finding Techniques, IEEE Transactions on Software Engineering, v.31 n.6, p.466-480, June 2005 [doi>10.1109/TSE.2005.63]
	22	Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
	23	Jiang Zheng , Laurie Williams , Nachiappan Nagappan , Will Snipes , John P. Hudepohl , Mladen A. Vouk, On the Value of Static Analysis for Fault Detection in Software, IEEE Transactions on Software Engineering, v.32 n.4, p.240-253, April 2006 [doi>10.1109/TSE.2006.38]
	24	Thomas Zimmermann , Rahul Premraj , Andreas Zeller, Predicting Defects for Eclipse, Proceedings of the Third International Workshop on Predictor Models in Software Engineering, p.9, May 20-26, 2007 [doi>10.1109/PROMISE.2007.10]