Generating a test oracle from program documentation: work in progress

ABSTRACT

A fundamental assumption of software testing is that there is some mechanism, an oracle, that will determine whether or not the results of a test execution are correct. In practice this is often done by comparing the output, either automatically or manually, to some pre-calculated, presumably correct, output [17]. However, if the program is formally documented it is possible to use the specification to determine the success or failure of a test execution, as in [1], for example. This paper discusses ongoing work to produce a tool that will generate a test oracle from formal program documentation.In [9], [10] and [11] Parnas et al. advocate the use of a relational model for documenting the intended behaviour of programs. In this method, tabular expressions are used to improve readability so that formal documentation can replace conventional documentation. Relations are described by giving their characteristic predicate in terms of the values of concrete program variables. This documentation method has the advantage that the characteristic predicate can be used as the test oracle -- it simply must be evaluated for each test execution (input & output) to assign pass or fail. In contrast to [1], this paper discusses the testing of individual programs, not objects as used in [1]. Consequently, the method works with program documentation, written in terms of the concrete variables, and no representation function need be supplied. Documentation in this form, and the corresponding oracle, are illustrated by an example.Finally, some of the implications of generating test oracles from relational specifications are discussed.

REFERENCES

• 1: Antoy, S. & Hamlet, R., "Objects that Check Themselves against Formal Specifications", TR 91-1, Dept. of Computer Science, Portland State University School of Engineering and Applied Sciences, Portland, OR. 18 pgs.
• 2: Gilles Bernot , Marie Claude Gaudel , Bruno Marre, Software testing based on formal specifications: a theory and a tool, Software Engineering Journal, v.6 n.6, p.387-405, Nov. 1991
• 3: David Chapman, A program testing assistant, Communications of the ACM, v.25 n.9, p.625-634, Sept 1982 [doi>10.1145/358628.358642]
• 4: John Gannon , Paul McMullin , Richard Hamlet, Data Abstraction, Implementation, Specification, and Testing, ACM Transactions on Programming Languages and Systems (TOPLAS), v.3 n.3, p.211-223, July 1981 [doi>10.1145/357139.357140]
• 5: Susan Gerhart , Dan Craigen , Ted Ralston, Experience with Formal Methods in Critical Systems, IEEE Software, v.11 n.1, p.21-28, January 1994 [doi>10.1109/52.251198]
• 6: Hamlet, R. G., "Testing Programs with the Aid of a Compiler", IEEE Transactions on Software Engineering, Vol. SE-3, No. 4 (July 1977), pp. 279~290.
• 7: David C. Luckham , Friedrich W. von Henke , Bernd Krieg-Brückner , Olaf Owe, ANNA: a language for annotating Ada programs, Springer-Verlag New York, Inc., New York, NY, 1987
• 8: Panzl, D. j., "Automatic Software Test Drivers", Computer, April 1978, pp. 44-50.
• 9: David Lorge Parnas, A generalized control structure and its formal definition, Communications of the ACM, v.26 n.8, p.572-581, Aug. 1983 [doi>10.1145/358161.358168]
• 10: Parnas, D.L. &Madey, J., "Functional Documentation for Computer Systems Engineering (Version 2)", CRL Report No. 237, Telecommunications Research Institute of Ontario (TRIO), September 1991, 14 pgs.
• 11: Parnas, D. L., Madey, J. & Iglewski, M., "Formal Documentation of Well-Structured Programs", CRL Report No. 259, Telecommunications Research institute of Ontario (TRIO), September 1992, 37 pgs.
• 12: Parnas, D. L., "Tabular Representation of Relations", CRL Report No. 260, Telecommunications Research Institute of Ontario (TRIO), November 1992, 17 pgs.
• 13: D. L. Parnas, Predicate Logic for Software Engineering, IEEE Transactions on Software Engineering, v.19 n.9, p.856-862, September 1993 [doi>10.1109/32.241769]
• 14: Peters, D. K., "Shortest Path Algorithm - Formal Program Documentation", CRL Report No. 280, Telecommunications Research Institute of Ontario (TRIO), February 1994, 11 pgs.
• 15: Debra J. Richardson , Stephanie Leif Aha , T. Owen O'Malley, Specification-based test oracles for reactive systems, Proceedings of the 14th international conference on Software engineering, p.105-118, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143100]
• 16: P. Stocks , D. Carrington, Test template framework: a specification-based testing case study, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.11-18, June 28-30, 1993, Cambridge, Massachusetts, United States
• 17: Weyuker, E. J., "On Testing Non-testable Programs", The Computer Journal, Vol. 25, No. 4 (1982), pp. 465- 470.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Methodologies (e.g., object-oriented, structured)
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Documentation
  D.3 PROGRAMMING LANGUAGES
      D.3.2 Language Classifications

          Nouns: C++

General Terms:
Algorithms, Design, Languages