Empirical evaluations of regression test selection techniques

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Empirical evaluations of regression test selection techniques: a systematic review

Full text

Pdf (257 KB)

Source

ESEM archive
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement table of contents

Kaiserslautern, Germany

SESSION: Testing and analysis table of contents

Pages 22-31

Year of Publication: 2008

ISBN:978-1-59593-971-5

Authors

Emelie Engström	Lund University, Lund, Sweden
Mats Skoglund	Lund University, Lund, Sweden
Per Runeson	Lund University, Lund, Sweden

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
ACM: Association for Computing Machinery

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 32, Downloads (12 Months): 326, Citation Count: 0

Additional Information:

abstract references index terms collaborative colleagues

Tools and Actions:	Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1414004.1414011 What is a DOI?

ABSTRACT

Regression testing is the verification that previously functioning software remains after a change. In this paper we report on a systematic review of empirical evaluations of regression test selection techniques, published in major software engineering journals and conferences. Out of 2,923 papers analyzed in this systematic review, we identified 28 papers reporting on empirical comparative evaluations of regression test selection techniques. They report on 38 unique studies (23 experiments and 15 case studies), and in total 32 different techniques for regression test selection are evaluated. Our study concludes that no clear picture of the evaluated techniques can be provided based on existing empirical evidence, except for a small group of related techniques. Instead, we identified a need for more and better empirical studies were concepts are evaluated rather than small variations. It is also necessary to carefully consider the context in which studies are undertaken.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
	2	Anjaneyulu Pasala Pasala , Animesh Bhowmick, An Approach for Test Suite Selection to Validate Applications on Deployment of COTS Upgrades, Proceedings of the 12th Asia-Pacific Software Engineering Conference, p.401-407, December 15-17, 2005 [doi>10.1109/APSEC.2005.31]
	3	John Bible , Gregg Rothermel , David S. Rosenblum, A comparative study of coarse- and fine-grained safe regression test-selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.149-183, April 2001 [doi>10.1145/367008.367015]
	4	Binkley, D. 1998. The application of program slicing to regression testing. Information and Software Technology. 40(11-12), 583--94.
	5	Pearl Brereton , Barbara A. Kitchenham , David Budgen , Mark Turner , Mohamed Khalil, Lessons from applying the systematic literature review process within the software engineering domain, Journal of Systems and Software, v.80 n.4, p.571-583, April, 2007 [doi>10.1016/j.jss.2006.07.009]
	6	Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
	7	Chengying Mao , Yansheng Lu, Regression Testing for Component-based Software Systems by Enhancing Change Information, Proceedings of the 12th Asia-Pacific Software Engineering Conference, p.611-618, December 15-17, 2005 [doi>10.1109/APSEC.2005.95]
	8	Tore Dyba , Torgeir Dingsoyr , Geir K. Hanssen, Applying Systematic Reviews to Diverse Study Types: An Experience Report, Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, p.225-234, September 20-21, 2007 [doi>10.1109/ESEM.2007.21]
	9	Elbaum, S., Kallakuri, P., Malishevsky, A., Rothermel, G., and Kanduri, S. 2003. Understanding the effects of changes on the cost-effectiveness of regression testing techniques. Software Testing, Verification and Reliability. 13(2), 65--83.
	10	Fischer, K., Raji, F., and Chruscicki, A. 1981. A methodology for retesting modified software. In NTC '81. IEEE 1981 National Telecommunications Conference. Innovative Telecommunications - Key to the Future. IEEE, 6-3.
	11	Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.184-208, April 2001 [doi>10.1145/367008.367020]
	12	Gregg, R. and Mary Jean, H. 2000. A Safe, Efficient Regression Test Selection Technique.
	13	Gupta, R., Harrold, M. J., and Soffa, M. L. 1992. An approach to regression testing using slicing. In Conference on Software Maintenance 1992 (Cat. No. 92CH3206-0). IEEE Comput. Soc. Press, 299--308.
	14	Gupta, R., Harrold, M. J., and Soffa, M. L. 1996. Program slicing-based regression testing techniques. Software Testing, Verification and Reliability. 6(2), 83--111.
	15	Florian Haftmann , Donald Kossmann , Eric Lo, A framework for efficient regression tests on database applications, The VLDB Journal — The International Journal on Very Large Data Bases, v.16 n.1, p.145-164, January 2007 [doi>10.1007/s00778-006-0028-8]
	16	Mary Jean Harrold , James A. Jones , Tongyu Li , Donglin Liang , Alessandro Orso , Maikel Pennings , Saurabh Sinha , S. Alexander Spoon , Ashish Gujarathi, Regression test selection for Java software, ACM SIGPLAN Notices, v.36 n.11, p.312-326, 11/01/2001
	17	Harrold, M. J. and Souffa, M. L. 1988. An incremental approach to unit testing during maintenance. In Proceedings of the Conference on Software Maintenance - 1988 (IEEE Cat. No. 88CH2615-3). IEEE Comput. Soc. Press, 362--7.
	18	Hartmann, J. and Robson, D. J. 1988. Approaches to regression testing. In Proceedings of the Conference on Software Maintenance - 1988 (IEEE Cat. No. 88CH2615-3). IEEE Comput. Soc. Press, 368--72.
	19	Jean Hartmann , David J. Robson, Techniques for Selective Revalidation, IEEE Software, v.7 n.1, p.31-36, January 1990 [doi>10.1109/52.43047]
	20	Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
	21	Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Infrastructure Support for Controlled Experimentation with Software Testing and Regression Testing Techniques, Proceedings of the 2004 International Symposium on Empirical Software Engineering, p.60-70, August 19-20, 2004 [doi>10.1109/ISESE.2004.23]
	22	Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, Applying regression test selection for COTS-based applications, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134357]
	23	Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, A Lightweight Process for Change Identification and Regression Test Selection in Using COTS Components, Proceedings of the Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, p.137, February 13-16, 2006 [doi>10.1109/ICCBSS.2006.1]
	24	Natalia Juristo , Ana M. Moreno , Sira Vegas , Martin Solari, In Search of What We Experimentally Know about Unit Testing, IEEE Software, v.23 n.6, p.72-80, November 2006 [doi>10.1109/MS.2006.166]
	25	Vigdis By Kampenes , Tore Dybå , Jo E. Hannay , Dag I. K. Sjøberg, Systematic review: A systematic review of effect size in software engineering experiments, Information and Software Technology, v.49 n.11-12, p.1073-1086, November, 2007 [doi>10.1016/j.infsof.2007.02.015]
	26	Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test application frequency: Research Articles, Software Testing, Verification & Reliability, v.15 n.4, p.257-279, December 2005 [doi>10.1002/stvr.v15:4]
	27	Kitchenham, B. A. 2007. Guidelines for performing Systematic Literature reviews in Software Engineering Version 2.3. Technical Report S.o.C.S.a.M. Software Engineering Group, Keele University and Department of Computer Science University of Durham.
	28	Barbara A. Kitchenham , Emilia Mendes , Guilherme H. Travassos, Cross versus Within-Company Cost Estimation Studies: A Systematic Review, IEEE Transactions on Software Engineering, v.33 n.5, p.316-329, May 2007 [doi>10.1109/TSE.2007.1001]
	29	René R. Klösch , Paul W. Glaser , Robert J. Truschnegg, A testing approach for large system portfolios in industrial environments, Journal of Systems and Software, v.62 n.1, p.11-20, 1 May 2002 [doi>10.1016/S0164-1212(01)00131-5]
	30	Toshihiko Koju , Shingo Takada , Norihisa Doi, Regression Test Selection based on Intermediate Code for Virtual Machines, Proceedings of the International Conference on Software Maintenance, p.420, September 22-26, 2003
	31	Leung, H. K. N. and White, L. 1989. Insights into regression testing. In Conference on Software Maintenance. Publ by IEEE, Piscataway, NJ, USA, 60--69.
	32	Leung, H. K. N. and White, L. 1990. A study of integration testing and software regression at the integration level. In Proceedings. Conference on Software Maintenance 1990 (Cat. No. 90CH2921-5). IEEE Comput. Soc. Press, 290--301.
	33	Mansour, N., Bahsoon, R., and Baradhi, G. 2001. Empirical comparison of regression test selection algorithms. Journal of Systems and Software. 57(1), 79--90.
	34	Nashat Mansour , Khaled Fakih, Natural Optimization Algorithms for Optimal Regression Testing, Proceedings of the 21st International Computer Software and Applications Conference, p.511-514, August 11-15, 1997
	35	Memon, A. M. 2004. Using tasks to automate regression testing of GUIs. In IASTED International Conference on Artificial Intelligence and Applications - AIA 2004. ACTA Press, 477--82.
	36	Using Component Metacontent to Support the Regression Testing of Component-Based Software, Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01), p.716, November 07-09, 2001
	37	Alessandro Orso , Nanjuan Shi , Mary Jean Harrold, Scaling regression testing to large software systems, ACM SIGSOFT Software Engineering Notes, v.29 n.6, November 2004
	38	Anjaneyulu Pasala Pasala , Animesh Bhowmick, An Approach for Test Suite Selection to Validate Applications on Deployment of COTS Upgrades, Proceedings of the 12th Asia-Pacific Software Engineering Conference, p.401-407, December 15-17, 2005 [doi>10.1109/APSEC.2005.31]
	39	Pei Hsia , Xiaolin Li , David Chenho Kung , Chih-Tung Hsu , Liang Li , Yasufumi Toyoshima , Cris Chen, A technique for the selective revalidation of OO software, Journal of Software Maintenance: Research and Practice, v.9 n.4, p.217-233, July-Aug. 1997 [doi>10.1002/(SICI)1096-908X(199707/08)9:4<217::AID-SMR152>3.3.CO;2-U]
	40	Xiaoxia Ren , Fenil Shah , Frank Tip , Barbara G. Ryder , Ophelia Chesley, Chianti: a tool for change impact analysis of java programs, Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 24-28, 2004, Vancouver, BC, Canada
	41	Gregg Rothermel , Mary Jean Harrold, A framework for evaluating regression test selection techniques, Proceedings of the 16th international conference on Software engineering, p.201-210, May 16-21, 1994, Sorrento, Italy
	42	Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
	43	Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
	44	Rothermel, G., Harrold, M. J., and Dedhia, J. 2000. Regression test selection for C++ software. Journal of Software Testing Verification and Reliability. 10(2), 77--109.
	45	A. S. M. Sajeev , Bugi Wibowo, Regression Test Selection Based on Version Changes of Components, Proceedings of the Tenth Asia-Pacific Software Engineering Conference Software Engineering Conference, p.78, December 10-12, 2003
	46	Skoglund, M. and Runeson, P. 2005. A case study of the class firewall regression test selection technique on a large scale distributed software system. In 2005 International Symposium on Empirical Software Engineering (IEEE Cat. No. 05EX1213). IEEE, 10 pp.
	47	Mark Staples , Mahmood Niazi, Experiences using systematic review guidelines, Journal of Systems and Software, v.80 n.9, p.1425-1437, September, 2007 [doi>10.1016/j.jss.2006.09.046]
	48	Toshihiko Koju , Shingo Takada , Norihisa Doi, Regression Test Selection based on Intermediate Code for Virtual Machines, Proceedings of the International Conference on Software Maintenance, p.420, September 22-26, 2003
	49	White, L. and Abdullah, K. 1997. A firewall approach for the regression testing of object-oriented software. Software Quality Week
	50	Lee White , Khaled Jaber , Brian Robinson, Utilization of Extended Firewall for Object-Oriented Regression Testing, Proceedings of the 21st IEEE International Conference on Software Maintenance, p.695-698, September 25-30, 2005 [doi>10.1109/ICSM.2005.101]
	51	White, L. J. and Leung, H. K. N. 1992. A firewall concept for both control-flow and data-flow in regression integration testing. In Conference on Software Maintenance 1992 (Cat. No. 92CH3206-0). IEEE Comput. Soc. Press, 262--71.
	52	David Willmor , Suzanne M. Embury, A Safe Regression Test Selection Technique for Database-Driven Applications, Proceedings of the 21st IEEE International Conference on Software Maintenance, p.421-430, September 25-30, 2005 [doi>10.1109/ICSM.2005.15]
	53	F. I. Vokolos , P. G. Frankl, Pythia: a regression test selection tool based on textual differencing, IFIP TC5 WG5.4 3rd internatinal conference on on Reliability, quality and safety of software-intensive systems, p.3-21, January 1997, Athens, Greece
	54	F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
	55	W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997
	56	W. Eric Wong , Joseph R. Horgan , Saul London , Hira Agrawal Bellcore, A Study of Effective Regression Testing in Practice, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.264, November 02-05, 1997
	57	Ye Wu , Mei-Hwa Chen , Howard M. Kao, Regression Testing on Object-Oriented Programs, Proceedings of the 10th International Symposium on Software Reliability Engineering, p.270, November 01-04, 1999
	58	Yanping Chen , Robert L. Probert , D. Paul Sims, Specification-based regression test selection with risk analysis, Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research, p.1, September 30-October 03, 2002, Toronto, Ontario, Canada
	59	Yih-Farn Chen , David S. Rosenblum , Kiem-Phong Vo, TestTube: a system for selective regression testing, Proceedings of the 16th international conference on Software engineering, p.211-220, May 16-21, 1994, Sorrento, Italy
	60	Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, An Initial Study of a Lightweight Process for Change Identification and Regression Test Selection When Source Code Is Not Available, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, p.225-234, November 08-11, 2005 [doi>10.1109/ISSRE.2005.8]
	61	Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, A Lightweight Process for Change Identification and Regression Test Selection in Using COTS Components, Proceedings of the Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems, p.137, February 13-16, 2006 [doi>10.1109/ICCBSS.2006.1]