Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression testing alone. Differential testing works by creating test suites for both the original system and the modified system and contrasting both versions of the system with these two suites.

Differential testing is made possible by recent advances in automated unit test generation. Furthermore, it makes automated test generators more useful because it abstracts away the interpretation and management of large volumes of tests by focusing on the changes between test suites.

In our preliminary empirical study of 3 subjects, differential testing discovered 21%, 34%, and 21% more behavior changes than regression testing alone.

REFERENCES

• 1: Acree, A. T., Budd, T. A., DeMillo, R. A., Lipton, R. J., and Sayward, F. G., Mutation analysis, School of Inform. and Comput. Sci., Georgia Instit. Technol., Atlanta, Tech. Rep. GIT-ICS-79/08, Sept. 1979.
• 2: Agitar website. http://www.agitar.com/.
• 3: Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
• 4: Robert V. Binder, Testing object-oriented systems: models, patterns, and tools, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 5: Marat Boshernitsan , Roongko Doong , Alberto Savoia, From daikon to agitator: lessons and challenges in building a commercial tool for developer testing, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146258]
• 6: Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact, Empirical Software Engineering, v.10 n.4, p.405-435, October 2005 [doi>10.1007/s10664-005-3861-2]
• 7: Michael D. Ernst , Jake Cockrell , William G. Griswold , David Notkin, Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Transactions on Software Engineering, v.27 n.2, p.99-123, February 2001 [doi>10.1109/32.908957]
• 8: Michael Feathers, Working Effectively with Legacy Code, Prentice Hall PTR, Upper Saddle River, NJ, 2004
• 9: Gamma, E., Beck, K., Junit, http://www.junit.org/.
• 10: Mary Jean Harrold , James A. Jones , Tongyu Li , Donglin Liang , Alessandro Orso , Maikel Pennings , Saurabh Sinha , S. Alexander Spoon , Ashish Gujarathi, Regression test selection for Java software, Proceedings of the 16th ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, p.312-326, October 14-18, 2001, Tampa Bay, FL, USA
• 11: W. E. Howden, Reliability of the Path Analysis Testing Strategy, IEEE Transactions on Software Engineering, v.2 n.3, p.208-215, May 1976 [doi>10.1109/TSE.1976.233816]
• 12: JTopas website. http://jtopas.sourceforge.net/jtopas/.
• 13: Junit Factory website. http://www.JunitFactory.com/
• 14: Wes Masri , Andy Podgurski , David Leon, An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows, IEEE Transactions on Software Engineering, v.33 n.7, p.454-477, July 2007 [doi>10.1109/TSE.2007.1020]
• 15: Xiaoxia Ren , Fenil Shah , Frank Tip , Barbara G. Ryder , Ophelia Chesley, Chianti: a tool for change impact analysis of java programs, Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 24-28, 2004, Vancouver, BC, Canada
• 16: David Saff , Shay Artzi , Jeff H. Perkins , Michael D. Ernst, Automatic test factoring for java, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101927]
• 17: Anthony J. H. Simons, A theory of regression testing for behaviourally compatible object types: Research Articles, Software Testing, Verification & Reliability, v.16 n.3, p.133-156, September 2006 [doi>10.1002/stvr.v16:3]
• 18: E J Weyuker, Axiomatizing software test data adequacy, IEEE Transactions on Software Engineering, v.12 n.12, p.1128-1138, Dec.1986
• 19: Hong Zhu , Patrick A. V. Hall , John H. R. May, Software unit test coverage and adequacy, ACM Computing Surveys (CSUR), v.29 n.4, p.366-427, Dec. 1997 [doi>10.1145/267580.267590]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Algorithms, Experimentation, Reliability, Theory, Verification

Keywords:
automated test generation, change impact analysis, differential testing, regression testing, software maintenance, test repair