Tailoring testing to a specific compiler—experiences

ABSTRACT

The testing of the Univac UCS-Pascal compiler is described. Tests were acquired from various sources, converted from existing tests, and developed in house. Test development and execution using the Univac Test Controller System is illustrated with examples. The experiences gained from this and other compiler testing efforts are described.

REFERENCES

• 1: COBOL COMPILER VALIDATION SYSTEM (CCVS), VERSION 4.0, USER'S GUIDE (IMPLEMENTATION DOCUMENTATION), Federal Compiler Testing Center, August 1980, available from National Technical Information Service as publication PB80-219900.
• 2: William James Cody, Software Manual for the Elementary Functions (Prentice-Hall series in computational mathematics), Prentice-Hall, Inc., Upper Saddle River, NJ, 1980
• 3: John V. Cugini, Joan S. Bowden, and Mark W. Skall, NBS MINIMAL BASIC TEST PROGRAMS—VERSION 2, USER'S MANUAL, VOLUME 1—DOCUMENTATION, NBS Special Publication 500-70/1 (November 1980), pp. 15-16.
• 4: John V. Cugini, SPECIFICATIONS AND TEST METHODS FOR NUMERIC ACCURACY IN PROGRAMMING LANGUAGE STANDARDS, NBS Special Publication 500-77, June 1981.
• 5: Paul R. Eggert, "Runtime Checking for ISO Standard Pascal", IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, Vol. 7, No. 4 (July 1981), pp. 447-448.
• 6: FORTRAN COMPILER VALIDATION SYSTEM (FCVS78), VERSION 1.0, DETAILED TEST SPECIFICATIONS, Federal Compiler Testing Center, November 1978, available from National Technical Information Service as publication AD-A062-038.
• 7: Charles N. Fischer and Richard J. LeBlanc, "The Implementation of Run-Time Diagnostics in Pascal", IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, Vol. 6, No. 4 (July 1980), pp. 313-319.
• 8: R.A. Freak and A.H.J. Sale, PASCAL VALIDATION SUITE—VERSION 3.0, Department of Information Science, University of Tasmania, GPO Box 252C, Hobart, Tasmania 7001, Australia. It can be obtained in North America from: Richard J. Cichelli, c/o ANPA Research Institute, Box 598, Easton, PA 18042. Phone [phone]. It can be obtained in Europe from: DR. B. Wichmann, National Physical Laboratory, Teddington, TW11 OLW, Middlesex, England. Phone [phone] EXT 3976.
• 9: John B. Goodenough, The Ada compiler validation capability, ACM SIGPLAN Notices, v.15 n.11, November 1980
• 10: John B. Goodenough, ADA COMPILER VALIDATION IMPLEMENTERS' GUIDE (Waltham, Mass.: SofTech, Inc., 1980), available from National Technical Information Service as publication AD-A091-760 (October, 1980).
• 11: Stephen R. Greenwood, MACRO: a programming language, ACM SIGPLAN Notices, v.14 n.12, p.80-91, December 1979 [doi>10.1145/954004.954010]
• 12: Dick Grune (ed.), THE REVISED MC ALGOL 68 TEST SET, IW-122/79-November (Amsterdam, The Netherlands: Mathematisch Centrum {Kruislaan 413, 1098 SJ Amsterdam, The Netherlands}, 1979).
• 13: H. C. Gyllstrom , R. C. Knippel , L. C. Ragland , K. E. Spackman, The universal compiling system, ACM SIGPLAN Notices, v.14 n.12, p.64-70, December 1979 [doi>10.1145/954004.954008]
• 14: Kathleen Jensen , Niklaus Wirth, Pascal-User Manual and Report, Springer-Verlag New York, Inc., Secaucus, NJ, 1975
• 15: UNIVAC SERIES 1100 MACRO PROGRAMMER REFERENCE MANUAL, UP-8336.1, Sperry Univac, St. Paul, Minnesota (1981).
• 16: David V. Moffat, "Index to the Periodical Literature—1981 Pascal Bibliography (June, 1981)", SIGPLAN NOTICES, Vol. 16, No. 11 (November 1981), pp. 7-21.
• 17: ELEFUNT (FORTRAN ELEMENTARY FUNCTION TESTS), NESC Abstract 881 (Argonne, Illinois: National Energy Software Center, Argonne National Laboratory, 9700 South Cass Avenue—1980).
• 18: Paul Oliver, TRANSFERABILITY OF FORTRAN BENCHMARKS, available from National Technical Information Service as publication AD-AO39-741 (January 1975).
• 19: Paul Oliver, "Experiences in Building and Using Compiler Validation Systems", PROCEEDINGS OF NATIONAL COMPUTER CONFERENCE, AFIPS (1979), pp. 1051-1057.
• 20: G. David Ripley and Frederick C. Druseikis, "A Statistical Analysis of Syntax Errors", COMPUTER LANGUAGES, Vol. 3 (1978), pp. 227-240.
• 21: G. David Ripley, ERRONEOUS PASCAL CODE, David Sarnoff Research Center, Princeton, NJ 08540 (1981). This is a set of Pascal syntax error analysis and recovery tests.
• 22: G. David Ripley, PASCAL SYNTAX ERROR DATA, report accompanying test programs (1981).
• 23: R.S. Scowen and Z.J. Ciechanowicz, COMPILER VALIDATION—A SURVEY, NPL CSU Technical Report No 8/81, National Physical Laboratory, Teddington, Middlesex TW11 OLW, United Kingdom (December 1980).
• 24: Harlan K. Seyfer, "Compiler Test Sets", to appear in SIGPLAN NOTICES (1982).
• 25: SERIES 1100 SYSTEM TEST PACKAGE (STP) LEVEL 2R1, SRA-365. Sperry Univac Marketing, 3001 Metro Drive, Suite, 300, Minneapolis, Minnesota 55420 (1980).
• 26: USER GUIDE, SPERRY UNIVAC SERIES 1100 TEST CONTROLLER SYSTEM (TCS), LEVEL 2R1, RRD-A446.2, Sperry Univac Marketing, 3001 Metro Drive, Suite 300, Minneapolis, Minnesota 55420 (1980).
• 27: J. Welsh, W.J. Sneeringer and C.A.R. Hoare, "Ambiguities and Insecurities in Pascal", SOFTWARE—PRACTICE AND EXPERIENCE, Vol. 7 (1977), pp. 685-696.
• 28: B.A. Wichmann and B. Jones, "Testing Algol 60 Compilers", SOFTWARE PRACTICE AND EXPERIENCE, Vol. 6, No. 2 (April-June 1976), pp. 261-270.
• 29: B.A. Wichmann and A.H.J. Sale, A PASCAL PROCESSOR VALIDATION SUITE, Report CSU 7/80, (Teddington, England: National Physical Laboratory, 1980).

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.2 Language Classifications

          Nouns: Pascal
      D.3.4 Processors
          Subjects: Compilers

General Terms:
Verification