Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Regression testing is an expensive and frequently executed maintenance process used to revalidate modified software. To improve it, regression test selection (RTS) techniques strive to lower costs without overly reducing effectiveness by carefully selecting a subset of the test suite. Under certain conditions, some can even guarantee that the selected test cases perform no worse than the original test suite.But this ignores certain software development realities such as resource and time constraints that may prevent using RTS techniques as intended (e.g., regression testing must be done overnight, but RTS selection returns two days worth of tests). In practice, testers work around this by prioritizing the test cases and running only those that fit within existing constraints. Unfortunately this generally violates key RTS assumptions, voiding RTS technique guarantees and making regression testing performance unpredictable.Despite this, existing prioritization techniques are memoryless, implicitly assuming that local choices can ensure adequate long run performance. Instead, we proposed a new technique that bases prioritization on historical execution data. We conducted an experiment to assess its effects on the long run performance of resource constrained regression testing. Our results expose essential tradeoffs that should be considered when using these techniques over a series of software releases.

REFERENCES

• 1: Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
• 2: R.G. Brown, Statistical Forecasting for Inventory Control. New York: McGraw-Hill, 1959.
• 3: Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
• 4: K. Fischer, F. Raji, and A. Chruscicki. A methodology for retesting modified software. In Proc. of Nat'l. Tele. Conf. B-6-3, pages 1-6, Nov. 1981.
• 5: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, Proceedings of the 20th international conference on Software engineering, p.188-197, April 19-25, 1998, Kyoto, Japan
• 6: M.J. Harrold and M.L. Soffa. An incremental approach to unit testing during maintenance. In Proc. of the Conf. on Softw. Maint., pages 362-367, Oct. 1988.
• 7: Jean Hartmann , David J. Robson, Techniques for Selective Revalidation, IEEE Software, v.7 n.1, p.31-36, January 1990 [doi>10.1109/52.43047]
• 8: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 9: Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test application frequency, Proceedings of the 22nd international conference on Software engineering, p.126-135, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337196]
• 10: H.K.N. Leung and L.J. White. Insights into regression testing. In Proc. of Int'l. Conf. on Softw. Maint., pages 60-69, Oct. 1989.
• 11: H.K.N. Leung and L.J. White. A cost model to compare regression test strategies. In Proc. of Int'l. Conf. on Softw. Maint., pages 201-208, Oct. 1991.
• 12: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 13: T. Ostrand and E. Weyuker. Using dataflow analysis for regression testing. In Sixth Annual Pacific Northwest Softw. Qual. Conf., pages 233-247, Sept. 1988.
• 14: D. Rosenblum and G. Rothermel. A comparative study of regression test selection techniques. In Proc. of the 2nd Int'l. Workshop on Empir. Studies of Softw. Maint., Oct. 1997.
• 15: David S. Rosenblum , Elaine J. Weyuker, Lessons Learned from a Regression Testing Case Study, Empirical Software Engineering, v.2 n.2, p.188-191, 1997 [doi>10.1023/A:1009717821137]
• 16: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 17: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 18: G. Rothermel and M.J. Harrold. Aristotle: A system for research and development of program analysis based tools. Technical Report OSU-CISRC-3/97-TR17, The Ohio State University, Mar. 1997.
• 19: Gregg Rothermel , Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE Transactions on Software Engineering, v.24 n.6, p.401-419, June 1998 [doi>10.1109/32.689399]
• 20: Gregg Rothermel , Roland H. Untch , Chengyun Chu , Mary Jean Harrold, Test Case Prioritization: An Empirical Study, Proceedings of the IEEE International Conference on Software Maintenance, p.179, August 30-September 03, 1999
• 21: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
• 22: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
        D.2.13 Reusable Software
            Subjects: Reusable libraries

Additional Classification:
  G. Mathematics of Computing
  G.1 NUMERICAL ANALYSIS
      G.1.6 Optimization
          Subjects: Constrained optimization
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

General Terms:
Algorithms, Design, Performance

Keywords:
empirical study, prioritization, regression testing, test history