Finding failure-inducing changes in java programs using change classification

ABSTRACT

Testing and code editing are interleaved activities during program development. When tests fail unexpectedly, the changes that caused the failure(s) are not always easy to find. We explore how change classification can focus programmer attention on failure-inducing changes by automatically labeling changes Red, Yellow, or Green, indicating the likelihood that they have contributed to a test failure. We implemented our change classification tool JUnit/CIA as an ex-tension to the JUnit component within Eclipse, and evaluated its effectiveness in two case studies. Our results indicate that change classification is an effective technique for finding failure-inducing changes.

REFERENCES

• 1: Kent Beck, Extreme programming explained: embrace change, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 2: BOHNER, S.A., AND ARNOLD, R. S. An introduction to software change impact analysis. In Software Change Impact Analysis, S. A. Bohner and R. S. Arnold, Eds. IEEE Computer Society Press, 1996, pp. 1--26.
• 3: Ophelia C. Chesley , Xiaoxia Ren , Barbara G. Ryder, Crisp: A Debugging Tool for Java Programs, Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), p.401-410, September 25-30, 2005 [doi>10.1109/ICSM.2005.37]
• 4: Jong-Deok Choi , Andreas Zeller, Isolating failure-inducing thread schedules, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 5: Holger Cleve , Andreas Zeller, Locating causes of program failures, Proceedings of the 27th international conference on Software engineering, p.342-351, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062522]
• 6: DALLMEIER,V.,LINDIG, C., AND ZELLER, A. Lightweight defect localization for Java. In Proc. 19th European Conf. on Object- Oriented Programming (ECOOP'05) (Glasgow, Scotland, 2005).
• 7: Richard A. DeMillo , Hsin Pan , Eugene H. Spafford, Critical slicing for software fault localization, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, p.121-134, January 08-10, 1996, San Diego, California, United States
• 8: Michael Dean Ernst , David Notkin, Dynamically discovering likely program invariants, 2000
• 9: Neelam Gupta , Haifeng He , Xiangyu Zhang , Rajiv Gupta, Locating faulty code using failure-inducing chops, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101948]
• 10: Mary Jean Harrold , James A. Jones , Tongyu Li , Donglin Liang , Alessandro Orso , Maikel Pennings , Saurabh Sinha , S. Alexander Spoon , Ashish Gujarathi, Regression test selection for Java software, Proceedings of the 16th ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, p.312-326, October 14-18, 2001, Tampa Bay, FL, USA
• 11: Mary Jean Harrold , Gregg Rothermel , Rui Wu , Liu Yi, An empirical investigation of program spectra, Proceedings of the 1998 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.83-90, June 16-16, 1998, Montreal, Quebec, Canada
• 12: James A. Jones , Mary Jean Harrold, Empirical evaluation of the tarantula automatic fault-localization technique, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101949]
• 13: James A. Jones , Mary Jean Harrold , John Stasko, Visualization of test information to assist fault localization, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581397]
• 14: David Chenho Kung , Jerry Gao , Pei Hsia , F. Wen , Yasufumi Toyoshima , Cris Chen, Change Impact Identification in Object Oriented Software Maintenance, Proceedings of the International Conference on Software Maintenance, p.202-211, September 01, 1994
• 15: James Law , Gregg Rothermel, Whole program Path-Based dynamic impact analysis, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 16: Michelle Lee , A. Jefferson Offutt , Roger T. Alexander, Algorithmic Analysis of the Impacts of Changes to Object-Oriented Software, Proceedings of the Technology of Object-Oriented Languages and Systems (TOOLS 34'00), p.61, July 30-August 03, 2000
• 17: Ben Liblit , Alex Aiken , Alice X. Zheng , Michael I. Jordan, Bug isolation via remote program sampling, Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation, June 09-11, 2003, San Diego, California, USA
• 18: Ben Liblit , Mayur Naik , Alice X. Zheng , Alex Aiken , Michael I. Jordan, Scalable statistical bug isolation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 19: Chao Liu , Xifeng Yan , Long Fei , Jiawei Han , Samuel P. Midkiff, SOBER: statistical model-based bug localization, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 20: LYLE, J., AND WEISER, M. Automatic bug location by program slicing. In Proceedings of the Second International Conference on Computers and Applications (Beijing (Peking), China, 1987), pp. 877--883.
• 21: Alessandro Orso , Taweesup Apiwattanapong , Mary Jean Harrold, Leveraging field data for impact analysis and regression testing, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 22: Alessandro Orso , Taweesup Apiwattanapong , James Law , Gregg Rothermel , Mary Jean Harrold, An Empirical Comparison of Dynamic Impact Analysis Algorithms, Proceedings of the 26th International Conference on Software Engineering, p.491-500, May 23-28, 2004
• 23: REN, X., CHESLEY, O., AND RYDER, B. G. Crisp: A debugging tool for Java programs. IEEE Transactions on Software Engineering (April 2006). In press.
• 24: Xiaoxia Ren , Fenil Shah , Frank Tip , Barbara G. Ryder , Ophelia Chesley, Chianti: a tool for change impact analysis of java programs, Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 24-28, 2004, Vancouver, BC, Canada
• 25: RENIERIS, M., AND REISS, S. Fault localization with nearest neighbor queries. In In Proceedings of the 18th IEEE International Conference on Automated Software Engineering (Montreal, Quebec, Canada, October 2003), pp. 30--39.
• 26: Thomas Reps , Thomas Ball , Manuvir Das , James Larus, The use of program profiling for software maintenance with applications to the year 2000 problem, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.432-449, September 22-25, 1997, Zurich, Switzerland
• 27: Joseph R. Ruthruff , Margaret Burnett , Gregg Rothermel, An empirical study of fault localization for end-user programmers, Proceedings of the 27th international conference on Software engineering, p.352-361, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062523]
• 28: Barbara G. Ryder , Frank Tip, Change impact analysis for object-oriented programs, Proceedings of the 2001 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.46-53, June 2001, Snowbird, Utah, United States [doi>10.1145/379605.379661]
• 29: David Saff , Michael D. Ernst, Reducing wasted development time via continuous testing, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.281, November 17-21, 2003
• 30: David Saff , Michael D. Ernst, Mock object creation for test factoring, Proceedings of the ACM-SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, June 07-08, 2004, Washington DC, USA [doi>10.1145/996821.996838]
• 31: David Saff , Michael D. Ernst, Continuous testing in eclipse, Proceedings of the 27th international conference on Software engineering, p.668-669, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062600]
• 32: STOERZER, M., RYDER, B. G., REN, X., AND TIP, F. Finding failure-inducing changes using change classification. Tech. Rep. DCS-TR-582, Rutgers University Department of Computer Science, September 2005.
• 33: TIP, F. A survey of program slicing techniques. J. of Programming Languages 3, 3 (1995), 121--189.
• 34: Frank Tip , Peter F. Sweeney , Chris Laffra , Aldo Eisma , David Streeter, Practical extraction techniques for Java, ACM Transactions on Programming Languages and Systems (TOPLAS), v.24 n.6, p.625-666, November 2002 [doi>10.1145/586088.586090]
• 35: TONELLA, P. Using a concept lattice of decomposition slices for program understanding and impact analysis. IEEE Trans. on Softw. Engineering 29,6 (2003), 495--509.
• 36: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 37: Mark David Weiser, Program slices: formal, psychological, and practical investigations of an automatic program abstraction method, 1979
• 38: Andreas Zeller, Yesterday, my program worked. Today, it does not. Why?, Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering, p.253-267, September 06-10, 1999, Toulouse, France
• 39: Andreas Zeller, Making students read and review code, Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSEconference on Innovation and technology in computer science education, p.89-92, July 11-13, 2000, Helsinki, Finland
• 40: Andreas Zeller, Isolating cause-effect chains from computer programs, Proceedings of the 10th ACM SIGSOFT symposium on Foundations of software engineering, November 18-22, 2002, Charleston, South Carolina, USA [doi>10.1145/587051.587053]
• 41: Andreas Zeller , Ralf Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE Transactions on Software Engineering, v.28 n.2, p.183-200, February 2002 [doi>10.1109/32.988498]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Debugging aids

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Version control

General Terms:
Algorithms, Experimentation

Keywords:
change impact analysis, debugging, fault localization, testing, version control