Prioritizing test cases for regression testing

ABSTRACT

Test case prioritization techniques schedule test cases in an order that increases their effectiveness in meeting some performance goal. One performance goal, rate of fault detection, is a measure of how quickly faults are detected within the testing process; an improved rate of fault detection can provide faster feedback on the system under test, and let software engineers begin locating and correcting faults earlier than might otherwise be possible. In previous work, we reported the results of studies that showed that prioritization techniques can significantly improve rate of fault detection. Those studies, however, raised several additional questions: (1) can prioritization techniques be effective when aimed at specific modified versions; (2) what tradeoffs exist between fine granularity and coarse granularity prioritization techniques; (3) can the incorporation of measures of fault proneness into prioritization techniques improve their effectiveness? This paper reports the results of new experiments addressing these questions.

REFERENCES

• 1: I. S. Association. Software Engineering Standards, volume 3 of Std. 1061: Standard for Software Quality Methodology. Institute of Electrical and Electronics Engineers, 1999 edition, 1992.
• 2: Alberto Avritzer , Elaine J. Weyuker, The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software, IEEE Transactions on Software Engineering, v.21 n.9, p.705-716, September 1995 [doi>10.1109/32.464549]
• 3: A. L. Baker , J. M. Bieman , N. Fenton , D. A. Gustafson , A. Melton , R. Whitty, A philosophy for software measurement, Journal of Systems and Software, v.12 n.3, p.277-281, Jul. 1990 [doi>10.1016/0164-1212(90)90050-V]
• 4: M. Balcer , W. Hasling , T. Ostrand, Automatic generation of test scripts from formal test specifications, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.210-218, December 13-15, 1989, Key West, Florida, United States
• 5: Lionel C. Briand , Jürgen Wüst , Stefan V. Ikonomovski , Hakim Lounis, Investigating quality factors in object-oriented designs: an industrial case study, Proceedings of the 21st international conference on Software engineering, p.345-354, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302654]
• 6: M. E. Delamaro and J. C. Maldonado. Proteum{A Tool for the Assessment ofTest Adequacy for C Programs. In Proc. of the Conf. on Performability in Computing Sys. (PCS 96), pages 79{95, July 1996.
• 7: R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on Test Data Selection: Help for the Practicing Programmer. Computer, 11(4):34{41, Apr. 1978.
• 8: S. Elbaum, A. G. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. Technical Report 00-60-03, Oregon State University, Feb. 2000.
• 9: S. G. Elbaum and J. C. Munson. A standard for the measurement of C complexity attributes. Technical Report TR-CS-98-02, University of Idaho, Feb. 1998.
• 10: J. C. Munson , S. G. Elbaum, Code Churn: A Measure for Estimating the Impact of Code Change, Proceedings of the International Conference on Software Maintenance, p.24, March 16-19, 1998
• 11: Sebastian G. Elbaum , John C. Munson, Software Evolution and the Code Fault Introduction Process, Empirical Software Engineering, v.4 n.3, p.241-262, September 1999 [doi>10.1023/A:1009830727593]
• 12: Michael R. Garey , David S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, W. H. Freeman & Co., New York, NY, 1979
• 13: Tarak Goradia, Dynamic impact analysis: a cost-effective technique to enforce error-propagation, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.171-181, June 28-30, 1993, Cambridge, Massachusetts, United States
• 14: R. G. Hamlet. Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng., SE-3(4):279{290, July 1977.
• 15: Richard G. Hamlet, Probable correctness theory, Information Processing Letters, v.25 n.1, p.17-25, 20 April 1987 [doi>10.1016/0020-0190(87)90088-3]
• 16: M. Harrold and G. Rothermel. Aristotle: A system for research on and development of program analysis based tools. Technical Report OSU-CISRC- 3/97-TR17, Ohio State University, Mar 1997.
• 17: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 18: T. M. Khoshgoftaar and J. C. Munson. Predicting software development errors using complexity metrics. J. on Selected Areas in Comm., 8(2):253{261, Feb. 1990.
• 19: J. C. Munson. Software measurement: Problems and practice. Annals of Softw. Eng., 1(1):255{285, 1995.
• 20: J. C. Munson, S. G. Elbaum, R. M. Karcich, and J. P. Wilcox. Software risk assessment through software measurement and modeling. In Proc. IEEE Aerospace Conf., pages 137{147, Mar. 1998.
• 21: A. P. Nikora and J. C. Munson. Software evolution and the fault process. In Proc. Twenty Third Annual Softw. Eng. Workshop, NASA/Goddard Space Flight Center, 1998.
• 22: Akira K. Onoma , Wei-Tek Tsai , Mustafa Poonawala , Hiroshi Suganuma, Regression testing in an industrial environment, Communications of the ACM, v.41 n.5, p.81-86, May 1998 [doi>10.1145/274946.274960]
• 23: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 24: Gregg Rothermel , Roland H. Untch , Chengyun Chu , Mary Jean Harrold, Test Case Prioritization: An Empirical Study, Proceedings of the IEEE International Conference on Software Maintenance, p.179, August 30-September 03, 1999
• 25: Margaret C. Thompson , Debra J. Richardson , Lori A. Clarke, An information flow model of fault detection, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.182-192, June 28-30, 1993, Cambridge, Massachusetts, United States
• 26: Jeffrey M. Voas, PIE: A Dynamic Failure-Based Technique, IEEE Transactions on Software Engineering, v.18 n.8, p.717-727, August 1992 [doi>10.1109/32.153381]
• 27: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
• 28: W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Design, Experimentation, Measurement, Performance, Reliability, Theory