An empirical study of regression test application frequency

ABSTRACT

Regression testing is an expensive maintenance process used to revalidate modified software. Regression test selection (RTS) techniques try to lower the cost of regression testing by selecting and running a subset of the existing test cases. Many such techniques have been proposed and initial studies show that they can produce savings. We believe, however, that issues such as the frequency with which testing is done have a strong effect on the behavior of these techniques. Therefore, we conducted an experiment to assess the effects of test application frequency on the costs and benefits of regression test selection techniques. Our results expose essential tradeoffs that should be considered when using these techniques over a series of software releases.

REFERENCES

• 1: Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
• 2: Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
• 3: K. Fischer, F. Raji, and A. Chruscicki. A methodology for retesting modified software. In Proc. of Nat'l. Tele. Conf. B-6-3, pages 1-6, Nov. 1981.
• 4: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, Proceedings of the 20th international conference on Software engineering, p.188-197, April 19-25, 1998, Kyoto, Japan
• 5: M.J. Harrold and M.L. Soffa. An incremental approach to unit testing during maintenance. In Proc. of the Conf. on Softw. Maint., pages 362-367, Oct. 1988.
• 6: Jean Hartmann , David J. Robson, Techniques for Selective Revalidation, IEEE Software, v.7 n.1, p.31-36, January 1990 [doi>10.1109/52.43047]
• 7: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 8: H.K.N. Leung and L.J. White. A cost model to compare regression test strategies. In Proc. of Int'l. Conf. on Softw. Maint., pages 201-208. Oct. 1991.
• 9: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 10: T. Ostrand and E. Weyuker. Using dataflow analysis for regression testing. In Sixth Annual Pacific Northwest Softw. Qual. Conf., pages 233-247, Sept. 1988.
• 11: D. Rosenblum and G. Rothermel. A comparative study of regression test selection techniques. In Proc. of the 2 nd Int'l. Workshop on Empir. Studies of Softw. Maint., Oct. 1997.
• 12: David S. Rosenblum , Elaine J. Weyuker, Lessons Learned from a Regression Testing Case Study, Empirical Software Engineering, v.2 n.2, p.188-191, 1997 [doi>10.1023/A:1009717821137]
• 13: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 14: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 15: G. Rothermel and M.J. Harrold. Aristotle: A system for research and development of program analysis based tools. Technical Report OSU-CISRC-3/97-TR17, The Ohio State University, Mar. 1997.
• 16: Gregg Rothermel , Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE Transactions on Software Engineering, v.24 n.6, p.401-419, June 1998 [doi>10.1109/32.689399]
• 17: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

General Terms:
Design, Experimentation, Management, Measurement, Performance, Reliability, Theory, Verification

Keywords:
applicaton frequency, empirical study, regression testing