Theoretical comparison of testing methods

ABSTRACT

Comparison of software testing methods is meaningful only if sound theory relates the properties compared to actual software quality. Existing comparisons typically use anecdotal foundations with no necessary relationship to quality, comparing methods on the basis of technical terms the methods themselves define. In the most seriously flawed work, one method whose efficacy is unknown is used as a standard for judging other methods! Random testing, as a method that can be related to quality (in both the conventional sense of statistical reliability, and the more stringent sense of software assurance), offers the opportunity for valid comparison.

REFERENCES

• 1: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 2: T. A. Budd, The portable mutation testing suite, TR 83-8, Department of Computer Science, University of Arizona, March, 1983.
• 3: T. A. Budd and W. Miller, Testing numerical software, TR 83-18, Department of Computer Science, University of Arizona, November, 1983.
• 4: R. DeMillo, R. Lipton, and F. Sayward, Hints on test data selection: help for the practicing programmer, Computer 11 (April, 1978), 34-43.
• 5: R. A. DeMillo and A. Jefferson Offutt VI, Experimental results of automatically generated adequate test sets, Proceeding 6th Pacijic Northwest Sofrware Quality Conference, Portland, OR, September, 1988,210- 232.
• 6: J. Duran and S. Ntafos, An evaluation of random testing, IEEE Trans. Software Eng. SE- 10 (July, 1984), 438-444.
• 7: Gourlay, A mathematical framework for the investigation of testing, IEEE Trans. Software Eng. SE-9 (November, 1983), 786-709.
• 8: R. Hamlet, Testing programs with the aid of a compiler, IEEE Trans. on Software Eng. SE-3 (July, 1977), 279-290.
• 9: Richard G. Hamlet, Probable correctness theory, Information Processing Letters, v.25 n.1, p.17-25, 20 April 1987 [doi>10.1016/0020-0190(87)90088-3]
• 10: R. Hamlet and R. Taylor, Partition testing does not inspire confidence, Proceedings Second Workshop on Sofnvare Testing, Verification, and Analysis, Banff, Canada, July, 1988,206-215.
• 11: R. Hamlet, Introduction to special section on software testing, Communications of the ACM, v.31 n.6, p.662-667, June 1988 [doi>10.1145/62959.62962]
• 12: R. Hamlet, Unit testing for software assurance, Proceedings COMPASS 89, Washington, DC, June, 1989,42-48.
• 13: W. Howden, Reliability of the path analysis testing strategy, IEEE Trans. Sofnvare Eng. SE-2(1976), 208- 215.
• 14: William E Howden, Functional program testing and analysis, McGraw-Hill, Inc., New York, NY, 1986
• 15: J. La&i and B. Korel, A data flow oriented program testing strategy, IEEE Trans. Software Eng. SE-9 (May, 1983), 347-354.
• 16: L. Lauterbach and W. Randall, Experimental evaluation of six test techniques, Proceedings COMPASS 89, Washington, DC, June, 1989.3641.
• 17: IEEE Software Staff, Faults, Failures, and a Metrics Revolution, IEEE Software, v.6 n.2, p.85, March 1989
• 18: Simeon C. Ntafos, An evaluation of required element testing strategies, Proceedings of the 7th international conference on Software engineering, p.250-256, March 26-29, 1984, Orlando, Florida, United States
• 19: S. C. Ntafos, A Comparison of Some Structural Testing Strategies, IEEE Transactions on Software Engineering, v.14 n.6, p.868-874, June 1988 [doi>10.1109/32.6165]
• 20: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 21: D. L. Pamas, A. van Schouwn, and S. Kwan, Evaluation standards for safety critical software, TR 88-220, Department of Computing and Information Science, Queen's University, Kingston, Ontario, Canada.
• 22: D. Parnas, personal communication.
• 23: C. V. Ramamoorthy, S. F. Ho, and W. T. Chen, On the automated generation of program test data, IEEE Trans. Software. Eng. SE-2 (Dec., 1976), 293-300.
• 24: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 25: Debra J. Richardson , Lori A. Clarke, A partition analysis method to increase program reliability, Proceedings of the 5th international conference on Software engineering, p.244-253, March 09-12, 1981, San Diego, California, United States
• 26: Richard W. Selby , Victor R. Basili , F. Terry Baker, Cleanroom software development: an empirical evaluation, IEEE Transactions on Software Engineering, v.13 n.9, p.1027-1037, September 1, 1987 [doi>10.1109/TSE.1987.233525]
• 27: P. Thevenod-Fosse, Statistical validation by means of statistical testing, Dependable Computing for Critical Applications, Santa Barbara, CA, August, 1989.
• 28: M. Weiser, J. Gannon, and P. McMullin, Comparison of struchual test coverage metrics, IEEE Software (March, 1985),80-85.
• 29: E J Weyuker, Axiomatizing software test data adequacy, IEEE Transactions on Software Engineering, v.12 n.12, p.1128-1138, Dec.1986
• 30: S. J. Zeil, The EQUATE testing strategy, Proceedings Workshop on Sofnvare Testing, Banff, Canada, July, 1986.142-151.
• 31: S. H. Z. Zweben , J. S. Gourlay, Comments on 'Axiomatizing Software Test Data Adequacy' by E.J. Weyuker, IEEE Transactions on Software Engineering, v.15 n.4, p.496-500, April 1989 [doi>10.1109/32.16609]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

General Terms:
Management, Reliability, Theory