Applicability of Software Validation Techniques to Scientific Programs

ABSTRACT

Error analysis involves the examination of a collection of programs whose errors are known. Each error is analyzed and validation techniques which would discover the error are identified. The errors that were present in version five of a package of Fortran scientific subroutines and then later corrected in version six were analyzed. An integrated collection of static and dynamic analysis methods would have discovered the errors in version five before its release. An integrated approach to validation and the effectiveness of individual methods are discussed.

REFERENCES

• 1: BAKER, F.T. Structured programming in a production programming environment. IEEE Trans. Softw. Eng. SE-1, 2(1975), 241-252.
• 2: BICERSKIS, J., ET AL. SMOTL--A system to construct samples for data processing program debugging. In Software Testing, vol. 2, A.E. Westley (Ed.), Infotech International, Maidenhead, England, 1979, pp. 13-23.
• 3: B. W. Boehm , J. R. Brown , M. Lipow, Quantitative evaluation of software quality, Proceedings of the 2nd international conference on Software engineering, p.592-605, October 13-15, 1976, San Francisco, California, United States
• 4: BOEHM, B.W., MCLEAN, R.K., AND URFRIC, D.B. Some experience with automated aids to the design of large scale reliable software. IEEE Trans. Softw. Eng. SE-1, 1(1975), 125-133.
• 5: Robert S. Boyer , Bernard Elspas , Karl N. Levitt, SELECT—a formal system for testing and debugging programs by symbolic execution, Proceedings of the international conference on Reliable software, p.234-245, April 21-23, 1975, Los Angeles, California
• 6: BROWN, J.R., ET AL. Automated software quality assurance. In Program Test Methods, W.C. Hetzel (Ed.), Prentice-Hall, Englewood Cliffs, N.J., 1973, pp. 181-204.
• 7: BUDD, T.A., DE MZLLO, R.A., LIPTON, R.J., AND SAYWARD, F.G. The design of a prototype mutation system for program testing. In Proc. AFIPS 1978 NCC, AFIPS Press, Alexandria, Va., 1978, pp. 623-627.
• 8: CLARKE, L.A. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. SE-2, 3(1976), 215-222.
• 9: DALY, E.G. Management of software development. IEEE Trans. Softw. Eng. SE-3, 3(1977), 230-232.
• 10: M. E. Fagan, Design and code inspections to reduce errors in program development, IBM Systems Journal, v.38 n.2-3, p.258-287, 1999
• 11: FuJii, M.S. Independent verification of highly reliable programs. In Proc. COMPSAC-77, Chicago, Ill., pp. 38-44.
• 12: GILB, T. Software Metrics. Winthrop, Cambridge, Mass., 1977.
• 13: GOODENOUOH, J., AND GERHART, S.L. Toward a theory of test data selection. IEEE Trans. Softw. Eng. SE-1, 2(1975), 156-173.
• 14: HAMLET, R.G. Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. SE-3, 4(1977), 279-289.
• 15: William C. Hetzel, An experimental analysis of program verification methods., 1976
• 16: HOLTHOUSE, M.A. Experience with automated testing analysis. In Digest, Workshop on Software Testing and Test Documentation, Fort Lauderdale, Fla., 1978, pp. 309-323.
• 17: HOWDEN, W.E. An evaluation of the effectiveness of symbolic testing and of testing on actual data. Softw. Pract. Exper. 8, 4(1978), 381-397.
• 18: HOWDEN, W.E. A survey of dynamic analysis methods. In Software Testing and Validation Techniques, E. Miller and W.E. Howden (Eds.), IEEE, Long Beach, Calif., 1978, pp. 184-206.
• 19: HOWDEN, W.E. Functional program testing. IEEE Trans. Softw. Eng SE-6, 3(1980).
• 20: HOWDEN, W.E. Functional testing and design abstractions. J. Syst. Softw. To appear.
• 21: HOWOEN, W.E. Methodology for the generation of program test data. IEEE Trans. Comput. C-24, 5(1975), 554-560.
• 22: HOWDEN, W.E. Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans. Softw. Eng. SE-3, 4(1977), 266-278.
• 23: J. C. Huang, An Approach to Program Testing, ACM Computing Surveys (CSUR), v.7 n.3, p.113-128, Sept. 1975 [doi>10.1145/356651.356652]
• 24: HUANO, J.C. Program instrumentation. A tool for software testing. In Software Testing, vol. 2, A.E. Westley (Ed.), Infotech International, Maidenhead, England, 1979, pp. 147-160.
• 25: IMSL Library reference manual. International Mathematical and Statistical Libraries, Inc., Houston, Tex., 1978.
• 26: M. A. Jackson, Principles of Program Design, Academic Press, Inc., Orlando, FL, 1975
• 27: James C. King, Symbolic execution and program testing, Communications of the ACM, v.19 n.7, p.385-394, July 1976 [doi>10.1145/360248.360252]
• 28: KRAUSE, K.W., SMITH, D.W., ANO GOODWIN, M.A. Optimal software test planning through automated network analysis. In Proc. IEEE Symp. Computer Software Reliabililty, New York, 1973, pp. 18-22.
• 29: MILLER, E., ET AL. Structurally based automatic program testing. In Proc. EASCON' 74, 1974.
• 30: E. F. Miller, Jr. , R. A. Melton, Automated generation of testcase datasets, Proceedings of the international conference on Reliable software, p.51-58, April 21-23, 1975, Los Angeles, California
• 31: Isao Miyamoto, Toward an effective software reliability evaluation, Proceedings of the 3rd international conference on Software engineering, p.46-55, May 10-12, 1978, Atlanta, Georgia, United States
• 32: Glenford J. Myers, A controlled experiment in program testing and code walkthroughs/inspections, Communications of the ACM, v.21 n.9, p.760-768, Sept. 1978 [doi>10.1145/359588.359602]
• 33: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 34: OSTERWEIL, L.J., AND FOSDICK, L.D. DAVE--A validation error detection and documentation system for FORTRAN programs. Softw. Pract. Exper. 6, 4(1976), 473-486.
• 35: PANZL, D.J. Automatic software test drivers. Computer 11, 4(1978), 44-50.
• 36: PRO/TEST Data Generator. Synergetics Corp., Burlington, Mass.
• 37: RAMAMOORTHY, C.V., AND HO, S.F. Testing large software with automated software evaluation systems. IEEE Trans. Sofiw. Eng. SE-1, 1(1975), 46-58.
• 38: STUCKI, L.G. Automatic generation of self-metric software. In Proc. IEEE Symp. Computer Software Reliabilty, New York, 1973, pp. 94-100.
• 39: STUCKI, L.G. New directions in automated tools for improving software quality. In Current Trends in Programming Methodology, vol. 2, R.T. Yeh (Ed.), Prentice-Hall, Englewood Cliffs, N.J., 1977, pp. 80-111.
• 40: WHITE, L.J., AND COHEN, E.I. A domain strategy for program testing. In Software Testing, vol. 2, A.E. Westley (Ed.), Infotech International, Maidenhead, England, 1979, pp. 325-362.
• 41: WOODWARD, M.R., HENNEL, M.A., AND HEDLEY, G. Observations and experience of path analysis and testing of programs. In Digest, Workshop on Software Testing and Test Documentation, Fort Lauderdale, Fla., 1978, pp. 70-96.

INDEX TERMS

Primary Classification:
  J. Computer Applications

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Validation

General Terms:
Design, Languages, Performance, Theory