TAOS: Testing with Analysis and Oracle Support

ABSTRACT

Few would question that software testing is a necessary activity for assuring software quality, yet the typical testing process is a human intensive activity and as such, it is unproductive, error-prone, and often inadequately done. Moreover, testing is seldom given a prominent place in software development or maintenance processes, nor is it an integral part of them. Major productivity and quality enhancements can be achieved by automating the testing process through tool development and use and effectively incorporating it with development and maintenance processes. The TAOS toolkit, Testing with Analysis and Oracle Support, provides support for the testing process. It includes tools that automate many tasks in the testing process, including management and persistence of test artifacts and the relationships between those artifacts, test development, test execution, and test measurement. A unique aspect of TAOS is its support for test oracles and their use to verify behavioral correctness of test executions. TAOS also supports structural/dependence coverage, by measuring the adequacy of test criteria coverage, and regression testing, by identifying tests associated or dependent upon modified software artifacts. This is accomplished by integrating the ProDAG toolset, Program Dependence Analysis Graph, with TAOS, which supports the use of program dependence analysis in testing, debugging, and maintenance. This paper describes the TAOS toolkit and its capabilities as well as testing, debugging and maintenance processes based on program dependence analysis. We also describe our experience with the toolkit and discuss our future plans.

REFERENCES

• CR94: Juei Chang and Debra J. Richardson. Static and Dynamic Specification Slicing. In Proceedings of the Fourth Annual lrvine Software Symposium, pages 25- 37, April 1994.
• CRZ88: Lori A. Clarke , Debra J. Richardson , Steven J. Zeil, TEAM: a support environment for testing, evaluation, and analysis, Proceedings of the third ACM SIGSOFT/SIGPLAN software engineering symposium on Practical software development environments, p.153-162, November 28-30, 1988, Boston, Massachusetts, United States
• DKMS+92: L. K. Dillon , G. Kutty , L. E. Moser , P. M. Melliar-Smith , Y. S. Ramakrishna, Graphical specifications for concurrent software systems, Proceedings of the 14th international conference on Software engineering, p.214-224, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143116]
• FOW87: Jeanne Ferrante , Karl J. Ottenstein , Joe D. Warren, The program dependence graph and its use in optimization, ACM Transactions on Programming Languages and Systems (TOPLAS), v.9 n.3, p.319-349, July 1987 [doi>10.1145/24039.24041]
• HPR88: S. Horwitz , J. Prins , T. Reps, On the adequacy of program dependence graphs for representing programs, Proceedings of the 15th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.146-157, January 10-13, 1988, San Diego, California, United States [doi>10.1145/73560.73573]
• HPR89: Susan Horwitz , Jan Prins , Thomas Reps, Integrating noninterfering versions of programs, ACM Transactions on Programming Languages and Systems (TOPLAS), v.11 n.3, p.345-387, July 1989 [doi>10.1145/65979.65980]
• HR92: Susan Horwitz , Thomas Reps, The use of program dependence graphs in software engineering, Proceedings of the 14th international conference on Software engineering, p.392-411, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143156]
• HRB90: Susan Horwitz , Thomas Reps , David Binkley, Interprocedural slicing using dependence graphs, ACM Transactions on Programming Languages and Systems (TOPLAS), v.12 n.1, p.26-60, Jan. 1990 [doi>10.1145/77606.77608]
• HT86: Susan Horwitz , Tim Teitelbaum, Generating editing environments based on relations and attributes, ACM Transactions on Programming Languages and Systems (TOPLAS), v.8 n.4, p.577-608, Oct. 1986 [doi>10.1145/6465.6512]
• KCTT91: Rudolf K. Keller , Mary Cameron , Richard N. Taylor , Dennis B. Troup, User interface development and software environments: the Chiron-1 system, Proceedings of the 13th international conference on Software engineering, p.208-218, May 13-17, 1991, Austin, Texas, United States
• KKL+81: D. J. Kuck , R. H. Kuhn , D. A. Padua , B. Leasure , M. Wolfe, Dependence graphs and compiler optimizations, Proceedings of the 8th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.207-218, January 26-28, 1981, Williamsburg, Virginia [doi>10.1145/567532.567555]
• Kor87: Bogdan Korel, The program dependence graph in static program testing, Information Processing Letters, v.24 n.2, p.103-108, January 1987 [doi>10.1016/0020-0190(87)90102-5]
• LK83: Janusz W. Laski and Bogdan Korel. A data flow oriented program testing strategy. IEEE Transactions on Software Engineering, SE-9(3):347-354, May 1983.
• Mau90: Peter M. Maurer, Generating Test Data with Enhanced Context-Free Grammars, IEEE Software, v.7 n.4, p.50-55, July 1990 [doi>10.1109/52.56422]
• Nta84: Simeon C. Ntafos. On required element testing. IEEE Transactions on Software Engineering, SE-10(6):795- 803, November 1984.
• OF76: Leon J. Osterweil and Lloyd D. Fosdick. DAVE - a validation, error detection, and documentation system for FORTRAN programs. Software -- Practice ~ Experience, 6:473-486, 1976.
• OO84: Karl J. Ottenstein , Linda M. Ottenstein, The program dependence graph in a software development environment, ACM SIGPLAN Notices, v.19 n.5, p.177-184, May 1984
• PC90: A. Podgurski , L. A. Clarke, A Formal Model of Program Dependences and its Implications for Software Testing, Debugging, and Maintenance, IEEE Transactions on Software Engineering, v.16 n.9, p.965-979, September 1990 [doi>10.1109/32.58784]
• RAO92: Debra J. Richardson , Stephanie Leif Aha , T. Owen O'Malley, Specification-based test oracles for reactive systems, Proceedings of the 14th international conference on Software engineering, p.105-118, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143100]
• RB93a: Debra J. Richardson and Bach Bui. Prodag graphical user interface manual. UCI-ICS Technical Report TR- 93-12, Department of Information and Computer Science, University of California, Irvine, August 1993.
• RB93b: Debra J. Richardson and Bach Bui. Taos graphical user interface manual. UCi-ICS Technical Report TR-93- 11, Department of Information and Computer Science, University of California, Irvine, August 1993.
• ROM+93: Debts J. Richardson, T. Owen O'Malley, Cynthia Tittle Moore, Stephanie H. Leif Aha, and Debra A. Brodbeck. ProDAG: An application programmatic interface for program dependence analysis graphs. Technical Report UCi-93-10, Department of Information and Computer Science, University of California, 1993.
• ROMA92: Debra J. Richardson , T. Owen O'Malley , Cynthia Tittle Moore , Stephanie Leif Aha, Developing and integrating ProDAG in the Arcadia environment, Proceedings of the fifth ACM SIGSOFT symposium on Software development environments, p.109-119, December 09-11, 1992, Tyson's Corner, Virginia, United States
• RW82: Sandra Rapps , Elaine J. Weyuker, Data flow analysis techniques for test data selection, Proceedings of the 6th international conference on Software engineering, p.272-278, September 13-16, 1982, Tokyo, Japan
• TC93: Peri Tarr , Lori A. Clarke, PLEIADES: an object management system for software engineering environments, Proceedings of the 1st ACM SIGSOFT symposium on Foundations of software engineering, p.56-70, December 08-10, 1993, Los Angeles, California, United States
• TRC92: Margaret C. Thompson, Debts J. Richardson, and Lori A. Clarke. information flow transfer in the Relay model. Technical Report TR-92..39, Department of Information and Computer Science, University of California, May 1992.
• Wei82: Mark Weiser, Programmers use slices when debugging, Communications of the ACM, v.25 n.7, p.446-452, July 1982 [doi>10.1145/358557.358577]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

          Nouns: DEBUG

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques
      D.2.4 Software/Program Verification
          Subjects: Validation

  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Computations on discrete structures
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.3 Studies of Program Constructs

General Terms:
Design, Human Factors, Reliability, Verification