An optimized change-driven regression testing selection strategy for binary Java applications

ABSTRACT

Selective regression testing involves re-testing of software systems with a subset of the whole test suite to verify that modifications have not caused adverse impacts to existing functions complied with the requirements specifications. With the growing of globalization and individual testing services providers, many testing and development teams belong to different organizations, and often the testing teams only get a binary release of the application without access to its source code. This makes source code analysis based regression test selection strategy not applicable. Meanwhile source code analysis based approach has scalability problem for large applications, which hinders its wide application in industry. This paper presents an optimized regression testing selection strategy based on binary java file change analysis, through which the problems around binary java applications are avoided. Besides static regression test suite reduction and test prioritization factors, continuous and real time testing execution information are incorporated as fault-proneness indicator of the selected test cases to dynamically select and prioritize the regression test suites. In addition, the whole strategy is lightweight, making the regression test selection process more automated and effective. Experiments show that this strategy can guarantee the change point coverage, reveal faults quickly, and scale to industry-size regression testing scenarios under resource and time constraints.

REFERENCES

• 1: Mary Jean Harrold , James A. Jones , Tongyu Li , Donglin Liang , Alessandro Orso , Maikel Pennings , Saurabh Sinha , S. Alexander Spoon , Ashish Gujarathi, Regression test selection for Java software, Proceedings of the 16th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, p.312-326, October 14-18, 2001, Tampa Bay, FL, USA
• 2: Zheng Li , Mark Harman , Robert M. Hierons, Search Algorithms for Regression Test Case Prioritization, IEEE Transactions on Software Engineering, v.33 n.4, p.225-237, April 2007 [doi>10.1109/TSE.2007.38]
• 3: Sebastian Elbaum , Alexey G. Malishevsky , Gregg Rothermel, Prioritizing test cases for regression testing, Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis, p.102-112, August 21-24, 2000, Portland, Oregon, United States
• 4: M. J. Harrold, G. Rothermel, K. Sayre, R. Wu, L. Yi. An Experimental Study of Regression Test Selection Techniques. In Journal of Software Testing, Verification, and Reliability, 10(3), 2000.
• 5: Gregg Rothermel , Roland J. Untch , Chengyun Chu, Prioritizing Test Cases For Regression Testing, IEEE Transactions on Software Engineering, v.27 n.10, p.929-948, October 2001 [doi>10.1109/32.962562]
• 6: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.184-208, April 2001 [doi>10.1145/367008.367020]
• 7: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 8: http://www.eclipse.org/tptp/.
• 9: Sebastian Elbaum , Alexey G. Malishevsky , Gregg Rothermel, Test Case Prioritization: A Family of Empirical Studies, IEEE Transactions on Software Engineering, v.28 n.2, p.159-182, February 2002 [doi>10.1109/32.988497]
• 10: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering, p.264, November 02-05, 1997
• 11: Yanping Chen , Robert L. Probert , D. Paul Sims, Specification-based regression test selection with risk analysis, Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research, p.1, September 30-October 03, 2002, Toronto, Ontario, Canada
• 12: Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, A process for identifying changes when source code is not available, Proceedings of the second international workshop on Models and processes for the evaluation of off-the-shelf components, May 21-21, 2005, St. Louis, Missouri
• 13: Jiang Zheng, In regression testing selection when source code is not available, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101997]
• 14: Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, Applying regression test selection for COTS-based applications, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134357]
• 15: Samuel Bates , Susan Horwitz, Incremental program testing using program dependence graphs, Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.384-396, March 1993, Charleston, South Carolina, United States [doi>10.1145/158511.158694]
• 16: P. Benedusi, A. Cimitile, and U. De Carlini. Post-Maintenance Testing Based on Path Change Analysis. In Proc. Conf. Software Maintenance, 1988, 352--361.
• 17: M. J. Harrold and M. L. Soffa. An Incremental Data Flow Testing Tool. In Proc. Sixth Int'l Conf. Testing Computer Software, 1989.
• 18: Gregg Rothermel , Mary Jean Harrold, A Safe, Efficient Algorithm for Regression Test Selection, Proceedings of the Conference on Software Maintenance, p.358-367, September 01, 1993
• 19: http://java.sun.com/j2se/1.5.0/docs/tooldocs/windows/javap.html
• 20: http://www.hutchies.iconbar.com/bigm.html

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification

General Terms:
Reliability, Verification

Keywords:
binary Java application, regression testing selection, test case prioritization