Automatically repairing event sequence-based GUI test suites for regression testing

ABSTRACT

Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing differ from those of traditional software. When the structure of a GUI is modified, test cases from the original GUI's suite are either reusable or unusable on the modified GUI. Because GUI test case generation is expensive, our goal is to make the unusable test cases usable, thereby helping to retain the suite's event coverage. The idea of reusing these unusable (obsolete) test cases has not been explored before. This article shows that a large number of test cases become unusable for GUIs. It presents a new GUI regression testing technique that first automatically determines the usable and unusable test cases from a test suite after a GUI modification, then determines the unusable test cases that can be repaired so that they can execute on the modified GUI, and finally uses repairing transformations to repair the test cases. This regression testing technique along with four repairing transformations has been implemented. An empirical study for four open-source applications demonstrates that (1) this approach is effective in that many of the test cases can be repaired, and is practical in terms of its time performance, (2) certain types of test cases are more prone to becoming unusable, and (3) certain types of “dominator” events, when modified, make a large number of test cases unusable.

REFERENCES

• 1: Abbot. 2003. Abbot Java GUI Test Framework. Available online at http://abbot.sourceforge.net.
• 2: Hiralal Agrawal , Joseph R. Horgan , Edward W. Krauser , Saul London, Incremental Regression Testing, Proceedings of the Conference on Software Maintenance, p.348-357, September 01, 1993
• 3: Alfred V. Aho , Ravi Sethi , Jeffrey D. Ullman, Compilers: principles, techniques, and tools, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1986
• 4: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 5: Benedusi, P., Cimitile, A., and DeCarlini, U. 1988. Post-maintenance testing based on path change analysis. In Proceedings of the IEEE Conference on Software Maintenance. IEEE Computer Society Press, Los Alamitos, CA, 352--368.
• 6: David Binkley, Semantics Guided Regression Test Cost Reduction, IEEE Transactions on Software Engineering, v.23 n.8, p.498-516, August 1997 [doi>10.1109/32.624306]
• 7: Monty L. Hammontree , Jeffrey J. Hendrickson , Billy W. Hensley, Integrated data capture and analysis tools for research and testing on graphical user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.431-432, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142886]
• 8: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 9: Mary Jean Harrold , John D. McGregor , Kevin J. Fitzpatrick, Incremental testing of object-oriented class structures, Proceedings of the 14th international conference on Software engineering, p.68-80, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143093]
• 10: Mary Jean Harrold , Mary Lou Soffa, Interprocedual data flow testing, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.158-167, December 13-15, 1989, Key West, Florida, United States
• 11: Hicinbothom, J. H. and Zachary, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. SPECIAL SESSIONS: Demonstrations, vol. 2. ACM Press, New York, NY, 1042.
• 12: JUnitResources. 2005. JUnit, testing resources for extreme programming. Available online at http://junit.org/news/extension/gui/index.htm.
• 13: David J. Kasik , Harry G. George, Toward automatic generation of novice user test scripts, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.244-251, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238519]
• 14: Kepple, L. R. 1992. A new paradigm for cross-platform automated GUI testing. X Res. 3, 1 (June), 155--178.
• 15: Kepple, L. R. 1994. The black art of GUI testing. Dr. Dobb's J. Softw. Tools 19, 2 (Feb.), 40.
• 16: David C. Kung , Jerry Gao , Pei Hsia , Yasufumi Toyoshima , Cris Chen, On regression testing of object-oriented programs, Journal of Systems and Software, v.32 n.1, p.21-40, Jan. 1996 [doi>10.1016/0164-1212(95)00047-X]
• 17: Thomas Lengauer , Robert Endre Tarjan, A fast algorithm for finding dominators in a flowgraph, ACM Transactions on Programming Languages and Systems (TOPLAS), v.1 n.1, p.121-141, July 1979 [doi>10.1145/357062.357071]
• 18: Marick, B. 2002. Bypassing the GUI. Softw. Test. Qual. Eng. Mag. 2, 41--47.
• 19: Atif Memon , Ishan Banerjee , Adithya Nagarajan, GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing, Proceedings of the 10th Working Conference on Reverse Engineering, p.260, November 13-17, 2003
• 20: Atif Memon , Adithya Nagarajan , Qing Xie, Automating regression testing for evolving GUI software: Research Articles, Journal of Software Maintenance and Evolution: Research and Practice, v.17 n.1, p.27-64, January 2005 [doi>10.1002/smr.v17:1]
• 21: Atif Muhammed Memon , Mary Lou Soffa, A comprehensive framework for testing graphical user interfaces, 2001
• 22: Atif M. Memon, GUI Testing: Pitfalls and Process, Computer, v.35 n.8, p.87-88, August 2002 [doi>10.1109/MC.2002.1023795]
• 23: Atif M. Memon , Martha E. Pollack , Mary Lou Soffa, Automated test oracles for GUIs, Proceedings of the 8th ACM SIGSOFT international symposium on Foundations of software engineering: twenty-first century applications, p.30-39, November 06-10, 2000, San Diego, California, United States
• 24: Atif M. Memon , Martha E. Pollack , Mary Lou Soffa, Hierarchical GUI Test Case Generation Using Automated Planning, IEEE Transactions on Software Engineering, v.27 n.2, p.144-155, February 2001 [doi>10.1109/32.908959]
• 25: Atif M. Memon , Mary Lou Soffa, Regression testing of GUIs, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 26: Atif M. Memon , Mary Lou Soffa , Martha E. Pollack, Coverage criteria for GUI testing, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, September 10-14, 2001, Vienna, Austria
• 27: Atif M. Memon , Qing Xie, Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software, IEEE Transactions on Software Engineering, v.31 n.10, p.884-896, October 2005 [doi>10.1109/TSE.2005.117]
• 28: Brad A. Myers, User interface software tools, ACM Transactions on Computer-Human Interaction (TOCHI), v.2 n.1, p.64-103, March 1995 [doi>10.1145/200968.200971]
• 29: Akira K. Onoma , Wei-Tek Tsai , Mustafa Poonawala , Hiroshi Suganuma, Regression testing in an industrial environment, Communications of the ACM, v.41 n.5, p.81-86, May 1998 [doi>10.1145/274946.274960]
• 30: Thomas Ostrand , Aaron Anodide , Herbert Foster , Tarak Goradia, A visual test development environment for GUI systems, Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis, p.82-92, March 02-04, 1998, Clearwater Beach, Florida, United States
• 31: Carlos Pacheco , Shuvendu K. Lahiri , Michael D. Ernst , Thomas Ball, Feedback-Directed Random Test Generation, Proceedings of the 29th international conference on Software Engineering, p.75-84, May 20-26, 2007 [doi>10.1109/ICSE.2007.37]
• 32: Lori L. Pollock , Mary Lou Soffa, Incremental global reoptimization of programs, ACM Transactions on Programming Languages and Systems (TOPLAS), v.14 n.2, p.173-200, April 1992 [doi>10.1145/128861.128865]
• 33: RationalRobot. 2003. Rational Robot. Available online at http://www.rational.com.ar/tools/robot.html.
• 34: Rosenberg, D. 1993. User interface prototyping paradigms in the 90's. In Proceedings of the ACM INTERCHI'93 Conference on Human Factors in Computing Systems—Adjunct Proceedings. Tutorials. ACM Press, New York, NY, 231.
• 35: Rosenblum, D. and Rothermel, G. 1997. A comparative study of regression test selection techniques. In Proceedings of the IEEE Computer Society 2nd International Workshop on Empirical Studies of Software maintenance. IEEE Computer Society Press, Los Alamitos, CA, 89--94.
• 36: David S. Rosenblum, A Practical Approach to Programming With Assertions, IEEE Transactions on Software Engineering, v.21 n.1, p.19-31, January 1995 [doi>10.1109/32.341844]
• 37: David S. Rosenblum , Elaine J. Weyuker, Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies, IEEE Transactions on Software Engineering, v.23 n.3, p.146-156, March 1997 [doi>10.1109/32.585502]
• 38: Gregg Rothermel , Mary Jean Harrold, A Safe, Efficient Algorithm for Regression Test Selection, Proceedings of the Conference on Software Maintenance, p.358-367, September 01, 1993
• 39: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 40: Gregg Rothermel , Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE Transactions on Software Engineering, v.24 n.6, p.401-419, June 1998 [doi>10.1109/32.689399]
• 41: Gregg Rothermel , Mary Jean Harrold , Jeffery Ostrin , Christie Hong, An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites, Proceedings of the International Conference on Software Maintenance, p.34, March 16-19, 1998
• 42: Richard K. Shehady , Daniel P. Siewiorek, A Method to Automate User Interface Testing Using Variable Finite State Machines, Proceedings of the 27th International Symposium on Fault-Tolerant Computing (FTCS '97), p.80, June 25-27, 1997
• 43: Software Research, Inc., Capture-Replay Tool 2003. Software Research, Inc., Capture-Replay Tool. Available online at http://soft.com.
• 44: Jim Thatcher, Screen Reader/2—programmed access to the GUI, Proceedings of the 4th international conference on Computers for handicapped persons, p.76-88, September 1994, Vienna, Austria
• 45: Jeffrey Voas, How Assertions Can Increase Test Effectiveness, IEEE Software, v.14 n.2, p.118-119,122, March 1997 [doi>10.1109/52.582981]
• 46: Walworth, A. 1997. Java GUI testing. Dr. Dobb's J. Softw. Tools 22, 2 (Feb.), 30, 32, 34.
• 47: Lee J. White, Regression Testing of GUI Event Interactions, Proceedings of the 1996 International Conference on Software Maintenance, p.350-358, November 04-08, 1996
• 48: Lee White , Husain Almezen, Generating Test Cases for GUI Responsibilities Using Complete Interaction Sequences, Proceedings of the 11th International Symposium on Software Reliability Engineering (ISSRE'00), p.110, October 08-11, 2000
• 49: Lee White , Husain Almezen , Shivakumar Sastry, Firewall Regression Testing of GUI Sequences and their Interactions, Proceedings of the International Conference on Software Maintenance, p.398, September 22-26, 2003
• 50: Lee White , Khaled Jaber , Brian Robinson, Utilization of Extended Firewall for Object-Oriented Regression Testing, Proceedings of the 21st IEEE International Conference on Software Maintenance, p.695-698, September 25-30, 2005 [doi>10.1109/ICSM.2005.101]
• 51: WinRunner. 2003. Mercury Interactive WinRunner. Available online at http://www.mercuryinteractive.com/products/winrunner.
• 52: Walter I. Wittel, Jr. , Theodore G. Lewis, Integrating the MVC Paradigm into an Object-Oriented Framework to Accelerate GUI Application Development, Oregon State University, Corvallis, OR, 1991
• 53: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000
• 54: Qing Xie , Atif M. Memon, Model-Based Testing of Community-Driven Open-Source GUI Applications, Proceedings of the 22nd IEEE International Conference on Software Maintenance, p.145-154, September 24-27, 2006 [doi>10.1109/ICSM.2006.39]
• 55: Qing Xie , Atif M. Memon, Designing and comparing automated test oracles for GUI-based software applications, ACM Transactions on Software Engineering and Methodology (TOSEM), v.16 n.1, p.4-es, February 2007 [doi>10.1145/1189748.1189752]
• 56: Xun Yuan , Atif M. Memon, Using GUI Run-Time State as Feedback to Generate Test Cases, Proceedings of the 29th international conference on Software Engineering, p.396-405, May 20-26, 2007 [doi>10.1109/ICSE.2007.94]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

General Terms:
Reliability, Verification

Keywords:
Graphical user interfaces, regression testing, repairing test cases, test case management, test maintenance