Extraction of bug localization benchmarks from history

ABSTRACT

Researchers have proposed a number of tools for automatic bug localization. Given a program and a description of the failure, such tools pinpoint a set of statements that are most likely to contain the bug. Evaluating bug localization tools is a difficult task because existing benchmarks are limited in size of subjects and number of bugs. In this paper we present iBUGS, an approach that semiautomatically extracts benchmarks for bug localization from the history of a project. For ASPECTJ, we extracted 369 bugs, 223 out of these had associated test cases. We demonstrate the relevance of our dataset with a case study on the bug localization tool AMPLE

REFERENCES

• 1: Holger Cleve , Andreas Zeller, Locating causes of program failures, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062522]
• 2: V. Dallmeier, C. Lindig, and A. Zeller. Lightweight defect localization for java. In Proceedings of 19th European Conference on Object-Oriented Programming (ECOOP), pages 528--550, 2005.
• 3: V. Dallmeier and T. Zimmermann. Automatic extraction of bug localization benchmarks from history. Technical report, Saarland University, Saarbrücken, Germany, June 2007.
• 4: Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact, Empirical Software Engineering, v.10 n.4, p.405-435, October 2005 [doi>10.1007/s10664-005-3861-2]
• 5: Zhenmin Li , Yuanyuan Zhou, PR-Miner: automatically extracting implicit programming rules and detecting violations in large software code, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 6: Ben Liblit , Mayur Naik , Alice X. Zheng , Alex Aiken , Michael I. Jordan, Scalable statistical bug isolation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 7: Chao Liu , Xifeng Yan , Long Fei , Jiawei Han , Samuel P. Midkiff, SOBER: statistical model-based bug localization, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 8: S. Lu, Z. Li, F. Qin, L. Tan, P. Zhou, and Y. Zhou. Bugbench: Benchmarks for evaluating bug detection tools. In PLDI Workshop on the Evaluation of Software Defect Detection Tools, June 2005.
• 9: Ranjith Purushothaman , Dewayne E. Perry, Toward Understanding the Rhetoric of Small Source Code Changes, IEEE Transactions on Software Engineering, v.31 n.6, p.511-526, June 2005 [doi>10.1109/TSE.2005.74]
• 10: Thomas Reps , Thomas Ball , Manuvir Das , James Larus, The use of program profiling for software maintenance with applications to the year 2000 problem, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.432-449, September 22-25, 1997, Zurich, Switzerland
• 11: Jacek Śliwerski , Thomas Zimmermann , Andreas Zeller, When do changes induce fixes?, Proceedings of the 2005 international workshop on Mining software repositories, p.1-5, May 17-17, 2005, St. Louis, Missouri
• 12: J. Spacco, D. Hovemeyer, and W. Pugh. Bugbench: Benchmarks for evaluating bug detection tools. In PLDI Workshop on the Evaluation of Software Defect Detection Tools, June 2005.
• 13: Jaime Spacco , Jaymie Strecker , David Hovemeyer , William Pugh, Software repository mining with Marmoset: an automated programming project snapshot and testing system, Proceedings of the 2005 international workshop on Mining software repositories, p.1-5, May 17-17, 2005, St. Louis, Missouri
• 14: Jinlin Yang , David Evans , Deepali Bhardwaj , Thirumalesh Bhat , Manuvir Das, Perracotta: mining temporal API rules from imperfect traces, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134325]
• 15: Xiangyu Zhang , Neelam Gupta , Rajiv Gupta, Locating faults through automated predicate switching, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134324]
• 16: Thomas Zimmermann, Fine-grained processing of CVS archives with APFEL, Proceedings of the 2006 OOPSLA workshop on eclipse technology eXchange, p.16-20, October 22-23, 2006, Portland, Oregon [doi>10.1145/1188835.1188839]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Debugging aids

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing); Tracing
      D.2.7 Distribution, Maintenance, and Enhancement

General Terms:
Management, Measurement, Reliability

Keywords:
benchmarking, defect localization