Comparing test data adequacy criteria

ABSTRACT

Test data adequacy criteria have been compared in a multitude of ways in the literature, ranging from the relative difficulty of satisfying them to the relative probability that test sets that satisfy them will expose errors in programs. Each method of comparison gives rise to an ordering of criteria, many of which differ significantly from the others. We investigate the various methods of comparing criteria, and show how the induced orderings are related. There are presently no methods of comparison that are based on the cost of using criteria; we propose a formal model of cost comparison of criteria. We categorize methods of comparison as being satisfiability based, correctness based, or complexity based.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Statistical computing

General Terms:
Design