Applying regression test selection for COTS-based applications

ABSTRACT

ABB incorporates a variety of commercial-off-the-shelf (COTS) components in its products. When new releases of these components are made available for integration and testing, source code is often not provided. Various regression test selection processes have been developed and have been shown to be cost effectiveness. However, the majority of these test selection techniques rely on access to source code for change identification. In this paper we present the application of the lightweight Integrated - Black-box Approach for Component Change Identification (I-BACCI) Version 3 process that select regression tests for applications that use COTS components. Two case studies, examining a total of nine new component releases, were conducted at ABB on products written in C/C++ to determine the effectiveness of I-BACCI. The results of the case studies indicate this process can reduce the required number of regression tests at least 70% without sacrificing the regression fault exposure.

REFERENCES

• 1: K. Abdullah , J. Kimble , L. White, Correcting for unreliable regression integration testing, Proceedings of the International Conference on Software Maintenance, p.232, October 17-20, 1995
• 2: Taweesup Apiwattanapong , Alessandro Orso , Mary Jean Harrold, A Differencing Algorithm for Object-Oriented Programs, Proceedings of the 19th IEEE international conference on Automated software engineering, p.2-13, September 20-24, 2004 [doi>10.1109/ASE.2004.5]
• 3: Victor R. Basili , Barry Boehm, COTS-Based Systems Top 10 List, Computer, v.34 n.5, p.91-93, May 2001 [doi>10.1109/2.920618]
• 4: John Bible , Gregg Rothermel , David S. Rosenblum, A comparative study of coarse- and fine-grained safe regression test-selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.149-183, April 2001 [doi>10.1145/367008.367015]
• 5: Thomas H. Cormen , Clifford Stein , Ronald L. Rivest , Charles E. Leiserson, Introduction to Algorithms, McGraw-Hill Higher Education, 2001
• 6: Gao, J. and Wu, Y., "Testing Component-Based Software - Issues, Challenges, and Solutions," in 3rd International Conference on COTS-Based Software Systems, Redondo Beach, 2004.
• 7: Jerry Zayu Gao , Jacob Tsao , Ye Wu , Taso H.-S. Jacob, Testing and Quality Assurance for Component-Based Software, Artech House, Inc., Norwood, MA, 2003
• 8: Todd L. Graves , Mary Jean Harrold , Jung-Min Kim , Adam Porter , Gregg Rothermel, An empirical study of regression test selection techniques, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.2, p.184-208, April 2001 [doi>10.1145/367008.367020]
• 9: Using Component Metacontent to Support the Regression Testing of Component-Based Software, Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01), p.716, November 07-09, 2001
• 10: IEEE, "IEEE Standard Glossary of Software Engineering Terminology," IEEE Standard 610.12, 1990.
• 11: David Chenho Kung , Jerry Gao , Pei Hsia , F. Wen , Yasufumi Toyoshima , Cris Chen, Change Impact Identification in Object Oriented Software Maintenance, Proceedings of the International Conference on Software Maintenance, p.202-211, September 01, 1994
• 12: Kung, D., Gao, J., Hsia, P., Wen, F., Toyoshima, Y., and Chen, C., "Class Firewall, Test Order and Regression Testing of Object-Oriented Programs," Journal of Object-Oriented Programming, Vol. 8, No. 2, 1995, pp. 51--65.
• 13: Laski, J. and Szermer, W., "Identification of program modifications and its applications in software maintenance," International Conference on Software Maintenance, 1992, pp. 282--290.
• 14: Leung, H. and White, L., "A Study of Integration Testing and Software Regression at the Integration Level," International Conference on Software Maintenance, San Diego, 1990, pp. 290--301.
• 15: H. K. N. Leung , L. White, Insights into testing and regression testing global variables, Journal of Software Maintenance: Research and Practice, v.2 n.4, p.209-222, Dec. 1990 [doi>10.1002/smr.4360020403]
• 16: Anneliese von Mayrhauser , Richard T. Mraz , Jeff Walls, Domain Based Regression Testing, Proceedings of the International Conference on Software Maintenance, p.26-35, September 01, 1994
• 17: Memon, A. M., "A process and role-based taxonomy of techniques to make testable COTS components," in Testing Commercial-off-the-shelf Components and Systems, S. Beydeda and V. Gruhn, Eds. Berlin, Germany: Springer-Verlag, 2005, pp. 109--140.
• 18: Alessandro Orso , Taweesup Apiwattanapong , James Law , Gregg Rothermel , Mary Jean Harrold, An Empirical Comparison of Dynamic Impact Analysis Algorithms, Proceedings of the 26th International Conference on Software Engineering, p.491-500, May 23-28, 2004
• 19: Xiaoxia Ren , Barbara G. Ryder , Maximilian Stoerzer , Frank Tip, Chianti: a change impact analysis tool for java programs, Proceedings of the 27th international conference on Software engineering, p.664-665, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062598]
• 20: Gregg Rothermel , Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE Transactions on Software Engineering, v.22 n.8, p.529-551, August 1996 [doi>10.1109/32.536955]
• 21: Srivastava, A., "Vulcan," TR-99-76, MSR 1999.
• 22: Amitabh Srivastava , Jay Thiagarajan, Effectively prioritizing tests in development environment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 23: F. I. Vokolos , P. G. Frankl, Pythia: a regression test selection tool based on textual differencing, IFIP TC5 WG5.4 3rd internatinal conference on on Reliability, quality and safety of software-intensive systems, p.3-21, January 1997, Athens, Greece
• 24: F. I. Vokolos , P. G. Frankl, Empirical Evaluation of the Textual Differencing Regression Testing Technique, Proceedings of the International Conference on Software Maintenance, p.44, March 16-19, 1998
• 25: Wang, Z., Pierce, K., and McFarling, S., "BMAT: A Binary Matching Tool for Stale Profile Propagation," The Journal of Instruction-Level Parallelism, Vol. 2, 2000.
• 26: Elaine J. Weyuker, Testing Component-Based Software: A Cautionary Tale, IEEE Software, v.15 n.5, p.54-59, September 1998 [doi>10.1109/52.714817]
• 27: White, L. and Abdullah, K., "A Firewall Approach for the Regression Testing of Object-Oriented Software," in Software Quality Week, San Francisco, 1997.
• 28: Lee White , Husain Almezen , Shivakumar Sastry, Firewall Regression Testing of GUI Sequences and their Interactions, Proceedings of the International Conference on Software Maintenance, p.398, September 22-26, 2003
• 29: White, L. and Leung, H., "A Firewall Concept for both Control-Flow and Data Flow in Regression Integration Testing," International Conference on Software Maintenance, Orlando, 1992, pp. 262--271.
• 30: Lee White , Brian Robinson, Industrial Real-Time Regression Testing and Analysis Using Firewalls, Proceedings of the 20th IEEE International Conference on Software Maintenance, p.18-27, September 11-14, 2004
• 31: Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, A process for identifying changes when source code is not available, Proceedings of the second international workshop on Models and processes for the evaluation of off-the-shelf components, May 21-21, 2005, St. Louis, Missouri
• 32: Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, A Lightweight Process for Change Identification and Regression Test Selection in Using COTS Components, Proceedings of the Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems (ICCBSS'05), p.137, February 13-16, 2006 [doi>10.1109/ICCBSS.2006.1]
• 33: Jiang Zheng , Brian Robinson , Laurie Williams , Karen Smiley, An Initial Study of a Lightweight Process for Change Identification and Regression Test Selection When Source Code Is Not Available, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, p.225-234, November 08-11, 2005 [doi>10.1109/ISSRE.2005.8]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification

General Terms:
Reliability, Verification

Keywords:
COTS, commercial-off-the-shelf, regression testing, software testing