Software unit test coverage and adequacy

ABSTRACT

Objective measurement of test quality is one of the key issues in software testing. It has been a major research focus for the last two decades. Many test criteria have been proposed and studied for this purpose. Various kinds of rationales have been presented in support of one criterion or another. We survey the research work in this area. The notion of adequacy criteria is examined together with its role in software dynamic testing. A review of criteria classification is followed by a summary of the methods for comparison and assessment of criteria.

REFERENCES

• 1: W. Richards Adrion , Martha A. Branstad , John C. Cherniavsky, Validation, Verification, and Testing of Computer Software, ACM Computing Surveys (CSUR), v.14 n.2, p.159-192, June 1982 [doi>10.1145/356876.356879]
• 2: Faten H. Afifi , Lee J. White , Steven J. Zeil, Testing for linear errors in nonlinear computer programs, Proceedings of the 14th international conference on Software engineering, p.81-91, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143096]
• 3: AMLA, N. AND AMMANN, P. 1992. Using Z specifications in category partition testing. In Proceedings of the Seventh Annual Conference on Computer Assurance (June), IEEE, 3-10.
• 4: AMMANN, P. AND OFFUTT, J. 1994. Using formal methods to derive test frames in categorypartition testing. In Proceedings of the Ninth Annual Conference on Computer Assurance (Gaithersburg, MD, June), IEEE, 69-79.
• 5: Richard Bache , Monika Mullerburg, Measures of testability as a basis for quality assurance, Software Engineering Journal, v.5 n.2, p.86-92, Mar. 1990
• 6: BAKER, A. L., HOWATT, J. W., AND BIEMAN, J. M. 1986. Criteria for finite sets of paths that characterize control flow. In Proceedings of the 19th Annual Hawaii International Conference on System Sciences, 158-163.
• 7: BASILI, V. R. AND RAMSEY, g. 1984. Structural coverage of functional testing. Tech. Rep. TR- 1442, Department of Computer Science, University of Maryland at College Park, Sept.
• 8: Victor R. Basili , Richard W. Selby, Comparing the effectiveness of software testing strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1, 1987 [doi>10.1109/TSE.1987.232881]
• 9: BAZZICHI, F. AND SPADAFORA, I. 1982. An automatic generator for compiler testing. IEEE Trans. Softw. Eng. SE-8, 4 (July), 343-353.
• 10: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 11: Boris Beizer, Software system testing and quality assurance, Van Nostrand Reinhold Co., New York, NY, 1984
• 12: N. M. Bengtson, Measuring errors in operational analysis assumptions, IEEE Transactions on Software Engineering, v.13 n.7, p.767-776, July 1987 [doi>10.1109/TSE.1987.233488]
• 13: BENTLY, W. G. AND MILLER, E.F. 1993. CT coverage-initial results. Softw. Quality J. 2, 1, 29-47.
• 14: Gilles Bernot , Marie Claude Gaudel , Bruno Marre, Software testing based on formal specifications: a theory and a tool, Software Engineering Journal, v.6 n.6, p.387-405, Nov. 1991
• 15: James M. Bieman , Janet L. Schultz, An empirical evaluation (and specification) of the all-du-paths testing criterion, Software Engineering Journal, v.7 n.1, p.43-51, Jan. 1992
• 16: David L Bird , Carlos Urias Munoz, Automatic generation of random self-checking test cases, IBM Systems Journal, v.22 n.3, p.229-245, 1983
• 17: L Bougé , N Choquet , L Fribourg , M C Gaudel, Test sets generation from algebraic specifications using logic programming, Journal of Systems and Software, v.6 n.4, p.343-360, Nov. 1986 [doi>10.1016/0164-1212(86)90004-X]
• 18: BUDD, T.A. 1981. Mutation analysis: Ideas, examples, problems and prospects. In Computer Program Testing, Chandrasekaran and Radicchi, Eds., North Holland, 129-148.
• 19: BUDD, T. A. AND ANGLUIN, D. 1982. Two notions of correctness and their relation to testing. Acta Inf. 18, 31-45.
• 20: BUDD, T. A., LIPTON, R. J., SAYWARD, F. G., AND DEMILLO, R.A. 1978. The design of a prototype mutation system for program testing. In Proceedings of National Computer Conference, 623-627.
• 21: Richard H. Carver , Kuo-Chung Tai, Replay and Testing for Concurrent Programs, IEEE Software, v.8 n.2, p.66-74, March 1991 [doi>10.1109/52.73751]
• 22: Jarir K. Chaar , Michael J. Halliday , Inderpal S. Bhandari , Ram Chillarege, In-Process Evaluation for Software Inspection and Test, IEEE Transactions on Software Engineering, v.19 n.11, p.1055-1070, November 1993 [doi>10.1109/32.256853]
• 23: CHANDRASEKARAN, B. AND RADICCHI, S. (EDS.) 1981. Computer Program Testing, North- Holland.
• 24: CHANG, C. C. AND KEISLER, H.J. 1973. Model Theory. North-Holland, Amsterdam.
• 25: Young-fu Chang , Mikio Aoyama, Guest Editor's Introduction: Testing the Limits of Test Technology, IEEE Software, v.8 n.2, p.9-11, March 1991
• 26: J. C. Cherniavsky , C. H. Smith, A recursion theoretic approach to program testing, IEEE Transactions on Software Engineering, v.13 n.7, p.777-784, July 1987 [doi>10.1109/TSE.1987.233489]
• 27: John C. Cherniavsky , Carl H. Smith, On Weyuker's Axioms for Software Complexity Measures, IEEE Transactions on Software Engineering, v.17 n.6, p.636-638, June 1991 [doi>10.1109/32.87287]
• 28: B. Choi , A. Mathur , B. Pattison, PMothra: scheduling mutants for execution on a hypercube, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.58-65, December 13-15, 1989, Key West, Florida, United States
• 29: T. Chusho, Test data selection and quality estimation based on the concept of essential branches for path testing, IEEE Transactions on Software Engineering, v.13 n.5, p.509-517, May 1987 [doi>10.1109/TSE.1987.233196]
• 30: CLARKE, L. A., HASSELL, J., AND RICHARDSON, D. J. 1982. A close look at domain testing. IEEE Trans. Softw. Eng. SE-8, 4 (July), 380-390.
• 31: L. A. Clarke , A. Podgurski , D. J. Richardson , S. J. Zeil, A Formal Evaluation of Data Flow Path Selection Criteria, IEEE Transactions on Software Engineering, v.15 n.11, p.1318-1332, November 1989 [doi>10.1109/32.41326]
• 32: P A Currit , M Dyer , H D Mills, Certifying the reliability of software, IEEE Transactions on Software Engineering, v.12 n.1, p.3-11, Jan. 1986
• 33: Martin Davis , Elaine Weyuker, Metric space-based test-base adequacy criteria, The Computer Journal, v.31 n.1, p.17-24, Feb., 1988 [doi>10.1093/comjnl/31.1.17]
• 34: DEMILLO, R. A. AND MATHUR, A. P. 1990. On the use of software artifacts to evaluate the effectiveness of mutation analysis for detecting errors in production software. In Proceedings of 13th Minnowbrook Workshop on Software Engineering (July 24-27, Blue Mountain Lake, NY), 75-77.
• 35: Richard A. DeMillo , A. Jefferson Offutt, Constraint-Based Automatic Test Data Generation, IEEE Transactions on Software Engineering, v.17 n.9, p.900-910, September 1991 [doi>10.1109/32.92910]
• 36: Richard A. DeMillo , A. Jefferson Offutt, Experimental results from an automatic test case generator, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.2, p.109-127, April 1993 [doi>10.1145/151257.151258]
• 37: DEMILLO, R. A., GUINDI, D. S., MCCRACKEN, W. M., OFFUTT, A. J., AND KING, K. N. 1988. An extended overview of the Mothra software testing environment. In Proceedings of SIG- SOFT Symposium on Software Testing, Analysis and Verification 2, (July), 142-151.
• 38: DEMILLO, R. A., LIPTON, R. J., AND SAYWARD, F.G. 1978. Hints on test data selection: Help for the practising programmer. Computer 11, (April), 34-41.
• 39: Richard A. DeMillo , W. Michael McCracken , R. J. Martin , John F. Passafiume, Software testing and evaluation, Benjamin-Cummings Publishing Co., Inc., Redwood City, CA, 1987
• 40: Richard Denney, Test-Case Generation from Prolog-Based Specifications, IEEE Software, v.8 n.2, p.49-57, March 1991 [doi>10.1109/52.73749]
• 41: DIJKSTRA, E.W. 1972. Notes on structured programming. In Structured Programming, by O.-J. Dahl, E. W. Dijkstra, and C. A. R. Hoare, Academic Press.
• 42: Thomas Downs, An approach to the modeling of software testing with some applications, IEEE Transactions on Software Engineering, v.11 n.4, p.375-386, April 1985 [doi>10.1109/TSE.1985.232227]
• 43: T Downs, Extensions to an approach to the modeling of software testing with some performance comparisons, IEEE Transactions on Software Engineering, v.12 n.9, p.979-987, Sept. 1986
• 44: DOWNS, T. AND GARRONE, P. 1991. Some new models of software testing with performance comparisons. IEEE Trans. Rel. 40, 3 (Aug.), 322-328.
• 45: I. M. M. Duncan , D. J. Robson, Ordered mutation testing, ACM SIGSOFT Software Engineering Notes, v.15 n.2, p.29-30, April 1990 [doi>10.1145/382296.382699]
• 46: DURAN, J. W. AND NTAFOS, S. 1984. An evaluation of random testing. IEEE Trans. Softw. Eng. SE-IO, 4 (July), 438-444.
• 47: Norman Fenton, When a software measure is not a measure, Software Engineering Journal, v.7 n.5, p.357-362, Sept., 1992
• 48: Norman E. Fenton, Software Metrics: A Rigorous Approach, Chapman & Hall, Ltd., London, UK, 1991
• 49: N. E. Fenton , R. W. Whitty , A. A. Kaposi, A generalised mathematical theory of structured programming, Theoretical Computer Science, v.36 n.2-3, p.145-171, June 1985
• 50: Ira R. Forman, An algebra for data flow anomaly detection, Proceedings of the 7th international conference on Software engineering, p.278-286, March 26-29, 1984, Orlando, Florida, United States
• 51: FOSTER, Z. n. 1980. Error sensitive test case analysis (ESTCA). IEEE Trans. Softw. Eng. SE-6, 3 (May), 258-264.
• 52: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 53: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194]
• 54: P. G. Frankl , E. J. Weyuker, A Formal Analysis of the Fault-Detecting Ability of Testing Methods, IEEE Transactions on Software Engineering, v.19 n.3, p.202-213, March 1993 [doi>10.1109/32.221133]
• 55: P. G. Frankl , E. J. Weyuker, Provable Improvements on Branch Testing, IEEE Transactions on Software Engineering, v.19 n.10, p.962-975, October 1993 [doi>10.1109/32.245738]
• 56: Roy S. Freedman, Testability of Software Components, IEEE Transactions on Software Engineering, v.17 n.6, p.553-564, June 1991 [doi>10.1109/32.87281]
• 57: Peter Fritzson , Tibor Gyimothy , Mariam Kamkar , Nahid Shahmehri, Generalized algorithmic debugging and testing, Proceedings of the ACM SIGPLAN 1991 conference on Programming language design and implementation, p.317-326, June 24-28, 1991, Toronto, Ontario, Canada
• 58: Susumu Fujiwara , Gregor von Bochmann , Ferhat Khendek , Mokhtar Amalou , Abderrazak Ghedamsi, Test Selection Based on Finite State Models, IEEE Transactions on Software Engineering, v.17 n.6, p.591-603, June 1991 [doi>10.1109/32.87284]
• 59: GAUDEL, M.-C. AND MARRE, B. 1988. Algebraic specifications and software testing: Theory and application. In Rapport LRI 407.
• 60: D. Gelperin , B. Hetzel, The growth of software testing, Communications of the ACM, v.31 n.6, p.687-695, June 1988 [doi>10.1145/62959.62965]
• 61: M. R. Girgis, An experimental evaluation of a symbolic execution system, Software Engineering Journal, v.7 n.4, p.285-290, July 1992
• 62: GOLD, E. M. 1967. Language identification in the limit. Inf. Cont. 10, 447-474.
• 63: GOODENOUGH, J. B. AND GERHART, S. L. 1975. Toward a theory of test data selection. IEEE Trans. Softw. Eng. SE-3 (June).
• 64: GOODENOUGH, J. B. AND GERHART, S. L. 1977. Toward a theory of testing: Data selection criteria. In Current Trends in Programming Methodology, Vol. 2, R. T. Yeh, Ed., Prentice- Hall, Englewood Cliffs, NJ, 44-79.
• 65: GOPAL, A. AND BUDD, T. 1983. Program testing by specification mutation. Tech. Rep. TR 83- 17, University of Arizona, Nov.
• 66: GOURLAY, J. 1983. A mathematical framework for the investigation of testing. IEEE Trans. Softw. Eng. SE-9, 6 (Nov.), 686-709.
• 67: P. A. V. Hall, Relationship between specifications and testing, Information and Software Technology, v.33 n.1, p.47-52, Jan./Feb. 1991 [doi>10.1016/0950-5849(91)90023-5]
• 68: HALL, P. A. V. AND HIERONS, R. 1991. Formal methods and testing. Tech. Rep. 91/16, Dept. of Computing, The Open University.
• 69: Dick Hamlet , Ross Taylor, Partition Testing Does Not Inspire Confidence (Program Testing), IEEE Transactions on Software Engineering, v.16 n.12, p.1402-1411, December 1990 [doi>10.1109/32.62448]
• 70: Dick Hamlet , Bruce Gifford , Borislav Nikolik, Exploring dataflow testing of arrays, Proceedings of the 15th international conference on Software Engineering, p.118-129, May 17-21, 1993, Baltimore, Maryland, United States
• 71: R. Hamlet, Theoretical comparison of testing methods, ACM SIGSOFT Software Engineering Notes, v.14 n.8, p.28-37, Dec. 1989
• 72: HAMLET, R.G. 1977. Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. 3, 4 (July), 279-290.
• 73: Mary Jean Harrold , John D. McGregor , Kevin J. Fitzpatrick, Incremental testing of object-oriented class structures, Proceedings of the 14th international conference on Software engineering, p.68-80, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143093]
• 74: M. Harrold , M. Soffa, Interprocedual data flow testing, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.158-167, December 13-15, 1989, Key West, Florida, United States
• 75: Mary Jean Harrold , Mary Lou Soffa, Selecting and Using Data for Integration Testing, IEEE Software, v.8 n.2, p.58-65, March 1991 [doi>10.1109/52.73750]
• 76: HARTWICK, D. 1977. Test planning. In Proceedings of National Computer Conference, 285- 294.
• 77: I J Hayes, Specification directed module testing, IEEE Transactions on Software Engineering, v.12 n.1, p.124-133, Jan. 1986
• 78: M. A. Hennell , D. Hedley , I. J. Riddell, Assessing a class of software tools, Proceedings of the 7th international conference on Software engineering, p.266-277, March 26-29, 1984, Orlando, Florida, United States
• 79: HERMAN, P. 1976. A data flow analysis approach to program testing. Aust. Comput. J. 8, 3 (Nov.), 92-96.
• 80: Bill Hetzel, The complete guide to software testing (2nd ed.), QED Information Sciences, Inc., Wellesley, MA, 1988
• 81: HIERONS, R. 1992. Software testing from formal specification. Ph.D. Thesis, Brunel University, UK.
• 82: Daniel M. Hoffman , Paul Strooper, Automated Module Testing in Prolog, IEEE Transactions on Software Engineering, v.17 n.9, p.934-943, September 1991 [doi>10.1109/32.92913]
• 83: J. R. Horgan , S. London, Data flow coverage and the C language, Proceedings of the symposium on Testing, analysis, and verification, p.87-97, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120815]
• 84: Joseph R. Horgan , Aditya P. Mathur, Assessing Testing Tools in Research and Education, IEEE Software, v.9 n.3, p.61-69, May 1992 [doi>10.1109/52.136170]
• 85: HOWDEN, W.E. 1975. Methodology for the generation of program test data. IEEE Trans. Comput. 24, 5 (May), 554-560.
• 86: HOWDEN, W. E. 1976. Reliability of the path analysis testing strategy. IEEE Trans. Softw. Eng. SE-2, (Sept.), 208-215.
• 87: HOWDEN, W.E. 1977. Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans. Softw. Eng. SE-3 (July), 266-278.
• 88: HOWDEN, W.E. 1978a. Algebraic program testing. ACTA Inf. 10, 53-66.
• 89: HOWDEN, W.E. 1978b. Theoretical and empirical studies of program testing. IEEE Trans. Softw. Eng. SE-4, 4 (July), 293-298.
• 90: HOWDEN, W.E. 1978c. An evaluation of the effectiveness of symbolic testing. Softw. Pract. Exper. 8, 381-397.
• 91: HOWDEN, W. E. 1980a. Functional program testing. IEEE Trans. Softw. Eng. SE-6, 2 (March), 162-169.
• 92: HOWDEN, W. E. 1980b. Functional testing and design abstractions. J. Syst. Softw. 1, 307-313.
• 93: William E. Howden, Completeness criteria for testing elementary program functions, Proceedings of the 5th international conference on Software engineering, p.235-243, March 09-12, 1981, San Diego, California, United States
• 94: William E. Howden, Validation of Scientific Programs, ACM Computing Surveys (CSUR), v.14 n.2, p.193-227, June 1982 [doi>10.1145/356876.356880]
• 95: HOWDEN, W. E. 1982b. Weak mutation testing and completeness of test sets. IEEE Trans. Softw. Eng. SE-8, 4 (July), 371-379.
• 96: HOWDEN, W.E. 1985. The theory and practice of functional testing. IEEE Softw. (Sept.), 6-17.
• 97: W E Howden, A functional approach to program testing and analysis, IEEE Transactions on Software Engineering, v.12 n.10, p.997-1005, Oct. 1986
• 98: William E Howden, Functional program testing and analysis, McGraw-Hill, Inc., New York, NY, 1986
• 99: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 100: INCE, D. C. 1987. The automatic generation of test data. Comput. J. 30, 1, 63-69.
• 101: INCE, D.C. 1991. Software testing. In Software Engineer's Reference Book, J. A. McDermid, Ed., Butterworth-Heinemann (Chapter 19).
• 102: Myron S. Karasik, Environmental testing techniques for software certification, IEEE Transactions on Software Engineering, v.11 n.9, p.934-938, Sept. 1985 [doi>10.1109/TSE.1985.232551]
• 103: KEMMERER, R.A. 1985. Testing formal specifications to detect design errors. IEEE Trans. Softw. Eng. SE-11, 1 (Jan.), 32-43.
• 104: Brian W. Kernighan , P. J. Plauger, Software Tools in Pascal, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1981
• 105: K. N. King , A. Jefferson Offutt, A Fortran language system for mutation-based software testing, Software—Practice & Experience, v.21 n.7, p.685-718, July 1991 [doi>10.1002/spe.4380210704]
• 106: B. Korel , H. Wedde , R. Ferguson, Dynamic method of test data generation for distributed software, Information and Software Technology, v.34 n.8, p.523-531, Aug. 1992 [doi>10.1016/0950-5849(92)90146-G]
• 107: KOSARAJU, S. 1974. Analysis of structured programs. J. Comput. Syst. Sci. 9, 232-255.
• 108: KRANTZ, D. H., LUCE, R. D., SUPPES, P., AND TVER- SKY, A. 1971. Foundations of Measurement, Vol. 1: Additive and Polynomial Representations. Academic Press, New York.
• 109: Edward W. Krauser , Aditya P. Mathur , Vernon J. Rego, High Performance Software Testing on SIMD Machines, IEEE Transactions on Software Engineering, v.17 n.5, p.403-423, May 1991 [doi>10.1109/32.90444]
• 110: Janusz Laski, Testing in the program development cycle, Software Engineering Journal, v.4 n.2, p.95-106, March 1989
• 111: LASKI, J. AND KOREL, B. 1983. A data flow oriented program testing strategy. IEEE Trans. Softw. Eng. SE-9, (May), 33-43.
• 112: Janusz Laski , Wojciech Szermer , Piotr Luczycki, Dynamic mutation testing in integrated regression analysis, Proceedings of the 15th international conference on Software Engineering, p.108-117, May 17-21, 1993, Baltimore, Maryland, United States
• 113: LAUTERBACH, L. AND RANDALL, W. 1989. Experimental evaluation of six test techniques. In Proceedings of COMPASS 89 (Washington, DC, June), 36-41.{
• 114: Ytzhak Levendel, Improving Quality With a Manufacturing Process, IEEE Software, v.8 n.2, p.13-25, March 1991 [doi>10.1109/52.73745]
• 115: LINDQUIST, T. E. AND JENKINS, g.R. 1987. Test case generation with IOGEN. In Proceedings of the 20th Annual Hawaii International Conference on System Sciences, 478-487.
• 116: Bev Littlewood , Lorenzo Strigini, Validation of ultrahigh dependability for software-based systems, Communications of the ACM, v.36 n.11, p.69-80, Nov. 1993 [doi>10.1145/163359.163373]
• 117: LIu, L.-L. AND ROBSON, D. J. 1989. Symbolic evaluation in software testing, the final report. Computer Science Tech. Rep. 10/89, School of Engineering and Applied Science, University of Durham, June.
• 118: LUCE, R. D., KRANTZ, D. H., SUPPES, P., AND TVER- SKY, A. 1990. Foundations of Measurement, Vol. 3: Representation, Axiomatization, and Invariance. Academic Press, San Diego.
• 119: Yashwant K. Malaiya , Anneliese von Mayrhauser , Pradip K. Srimani, An Examination of Fault Exposure Ratio, IEEE Transactions on Software Engineering, v.19 n.11, p.1087-1094, November 1993 [doi>10.1109/32.256855]
• 120: Brian Marick, The weak mutation hypothesis, Proceedings of the symposium on Testing, analysis, and verification, p.190-199, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120825]
• 121: MATHUR, A. P. 1991. Performance, effectiveness, and reliability issues in software testing. In Proceedings of the 15th Annual International Computer Software and Applications Conference (Tokyo, Sept.), 604-605.
• 122: MARSHALL, A. C. 1991. A Conceptual model of software testing. J. Softw. Test. Ver. Rel. 1, 3 (Dec.), 5-16.
• 123: MCCABE, T. g. 1976. A complexity measure. IEEE Trans. Softw. Eng. SE-2, 4, 308-320.
• 124: MCCABE, T. J. (ED.) 1983. Structured Testing. IEEE Computer Society Press, Los Alamitos, CA.
• 125: Thomas J. McCabe , G. G. Schulmeyer, System testing aided by structured analysis: a practical experience, IEEE Transactions on Software Engineering, v.11 n.9, p.917-921, Sept. 1985 [doi>10.1109/TSE.1985.232549]
• 126: MCMULLIN, P. R. AND GANNON, J.D. 1983. Combining testing with specifications: A case study. IEEE Trans. Softw. Eng. SE-9, 3 (May), 328-334.
• 127: MEEK, B. AND SIU, K.K. 1988. The effectiveness of error seeding. Alvey Project SE/064: Quality evaluation of programming language processors, Report No. 2, Computing Centre, King's College London, Oct.
• 128: MILLER, E. AND HOWDEN, W.E. 1981. Tutorial: Software Testing and Validation Techniques, (2nd ed.). IEEE Computer Society Press, Los Alamitos, CA.
• 129: Keith W. Miller , Larry J. Morell , Robert E. Noonan , Stephen K. Park , David M. Nicol , Branson W. Murrill , Jeffrey M. Voas, Estimating the Probability of Failure When Testing Reveals No Failures, IEEE Transactions on Software Engineering, v.18 n.1, p.33-43, January 1992 [doi>10.1109/32.120314]
• 130: L. J. Morell, A Theory of Fault-Based Testing, IEEE Transactions on Software Engineering, v.16 n.8, p.844-857, August 1990 [doi>10.1109/32.57623]
• 131: Glenford J. Myers, An extension to the cyclomatic measure of program complexity, ACM SIGPLAN Notices, v.12 n.10, p.61-64, October 1977 [doi>10.1145/954627.954633]
• 132: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 133: J. Paul Myers, Jr., The complexity of software testing, Software Engineering Journal, v.7 n.1, p.13-24, Jan. 1992
• 134: Simeon C. Ntafos, An evaluation of required element testing strategies, Proceedings of the 7th international conference on Software engineering, p.250-256, March 26-29, 1984, Orlando, Florida, United States
• 135: NTAFOS, S.C. 1984. On required element testing. IEEE Trans. Softw. Eng. SE-IO, 6 (Nov.), 795-803.
• 136: S. C. Ntafos, A Comparison of Some Structural Testing Strategies, IEEE Transactions on Software Engineering, v.14 n.6, p.868-874, June 1988 [doi>10.1109/32.6165]
• 137: A. Offutt, The coupling effect: fact or fiction, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.131-140, December 13-15, 1989, Key West, Florida, United States
• 138: A. Jefferson Offutt, Investigations of the software testing coupling effect, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.1, p.5-20, Jan. 1992 [doi>10.1145/125489.125473]
• 139: A. Jefferson Offutt , Stephen D. Lee, How strong is weak mutation?, Proceedings of the symposium on Testing, analysis, and verification, p.200-213, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120826]
• 140: A. Jefferson Offutt , Gregg Rothermel , Christian Zapf, An experimental evaluation of selective mutation, Proceedings of the 15th international conference on Software Engineering, p.100-107, May 17-21, 1993, Baltimore, Maryland, United States
• 141: Leon Osterweil , Lori A. Clarke, A Proposed Testing and Analysis Research Initiative, IEEE Software, v.9 n.5, p.89-96, September 1992 [doi>10.1109/52.156901]
• 142: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 143: Thomas J. Ostrand , Elaine J. Weyuker, Data flow-based test adequacy analysis for languages with pointers, Proceedings of the symposium on Testing, analysis, and verification, p.74-86, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120814]
• 144: Martyn A. Ould , Charles Unwin, Testing in software development, Cambridge University Press, New York, NY, 1987
• 145: PAIGE, M. R. 1975. Program graphs, an algebra, and their implication for programming. IEEE Trans. Softw. Eng. SE-1, 3, (Sept.), 286-291.
• 146: PAIGE, M. R. 1978. An analytical approach to software testing. In Proceedings COMP- SAC'78, 527-532.
• 147: Hemant D. Pande , William Landi, Interprocedural Def-Use associations in C programs, Proceedings of the symposium on Testing, analysis, and verification, p.139-153, October 08-10, 1991, Victoria, British Columbia, Canada [doi>10.1145/120807.120820]
• 148: Allen Parrish , Stuart H. Zweben, Analysis and Refinement of Software Test Data Adequacy Properties, IEEE Transactions on Software Engineering, v.17 n.6, p.565-581, June 1991 [doi>10.1109/32.87282]
• 149: A. S. Parrish , S. H. Zweben, Clarifying Some Fundamental Concepts in Software Testing, IEEE Transactions on Software Engineering, v.19 n.7, p.742-746, July 1993 [doi>10.1109/32.238573]
• 150: PETSCHENIK, N.H. 1985. Practical priorities in system testing. IEEE Softw. (Sept.), 18-23.
• 151: Paul Piwowarski , Mitsuru Ohba , Joe Caruso, Coverage measurement experience during function test, Proceedings of the 15th international conference on Software Engineering, p.287-301, May 17-21, 1993, Baltimore, Maryland, United States
• 152: A. Podgurski , L. Clarke, The implications of program dependencies for software testing, debugging, and maintenance, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.168-178, December 13-15, 1989, Key West, Florida, United States
• 153: A. Podgurski , L. A. Clarke, A Formal Model of Program Dependences and its Implications for Software Testing, Debugging, and Maintenance, IEEE Transactions on Software Engineering, v.16 n.9, p.965-979, September 1990 [doi>10.1109/32.58784]
• 154: R. E. Prather , J. P. Myers, Jr., The path prefix software testing strategy, IEEE Transactions on Software Engineering, v.13 n.7, p.761-766, July 1987 [doi>10.1109/TSE.1987.233487]
• 155: PROGRAM ANALYSIS LTD., UK. 1992. Testbed technical description. May.
• 156: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 157: RICHARDSON, D. J. AND CLARKE, L. A. 1985. Partition analysis: A method combining testing and verification. IEEE Trans. Softw. Eng. SE-11, 12 (Dec.), 1477-1490.{
• 158: Debra J. Richardson , Stephanie Leif Aha , T. Owen O'Malley, Specification-based test oracles for reactive systems, Proceedings of the 14th international conference on Software engineering, p.105-118, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143100]
• 159: RICHARDSON, D. J. AND THOMPSON, M. C. 1988. The RELAY model of error detection and its application. In Proceedings of SIG- SOFT Symposium on Software Testing, Analysis, and Verification 2 (July).
• 160: Debra J. Richardson , Margaret C. Thompson, An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection, IEEE Transactions on Software Engineering, v.19 n.6, p.533-553, June 1993 [doi>10.1109/32.232020]
• 161: RIDDELL, I. J., HENNELL, M. A., WOODWARD, M. R., AND HEDLEY, D. 1982. Practical aspects of program mutation. Tech. Rep., Dept. of Computational Science, University of Liverpool, UK.
• 162: ROBERTS, F.S. 1979. Measurement Theory, Encyclopedia of Mathematics and Its Applications, Vol. 7. Addison-Wesley, Reading, MA.
• 163: Robert P. Roe , John H. Rowland, Some theory concerning certification of mathematical subroutines by black box testing, IEEE Transactions on Software Engineering, v.13 n.6, p.677-682, June 1987 [doi>10.1109/TSE.1987.233205]
• 164: Nicolas Roussopoulos , Raymond T. Yeh, SEES—a software testing environment support system, IEEE Transactions on Software Engineering, v.11 n.4, p.355-366, April 1985 [doi>10.1109/TSE.1985.232225]
• 165: RUDNER, B. 1977. Seeding/tagging estimation of software errors: Models and estimates. Rome Air Development Centre, Rome, NY, RADC-TR-77-15, also AD-A036 655.
• 166: B. Sarikaya , G. v. Bochmann , E. Cerny, A test design methodology for protocol testing, IEEE Transactions on Software Engineering, v.13 n.5, p.518-531, May 1987 [doi>10.1109/TSE.1987.233197]
• 167: Susan A. Sherer, A Cost-Effective Approach to Testing, IEEE Software, v.8 n.2, p.34-40, March 1991 [doi>10.1109/52.73747]
• 168: SOFTWARE RESEARCH. 1992. Software Test- Works--Software Testers Workbench System. Software Research, Inc.
• 169: J. A. Solheim , J. H. Rowland, An Empirical Study of Testing and Integration Strategies Using Artificial Software Systems, IEEE Transactions on Software Engineering, v.19 n.10, p.941-949, October 1993 [doi>10.1109/32.245736]
• 170: P. A. Stocks , D. A. Carrington, Test templates: a specification-based testing framework, Proceedings of the 15th international conference on Software Engineering, p.405-414, May 17-21, 1993, Baltimore, Maryland, United States
• 171: Jason Su , Paul R. Ritter, Experience in Testing the Motif Interface, IEEE Software, v.8 n.2, p.26-33, March 1991 [doi>10.1109/52.73746]
• 172: SUPPES, P., KRANTZ, D. H., LUCE, R. D., AND TVER- SKY, A. 1989. Foundations of Measurement, Vol. 2: Geometrical, Threshold, and Probabilistic Representations. Academic Press, San Diego.
• 173: Kuo-Chung Tai, Predicate-based test generation for computer programs, Proceedings of the 15th international conference on Software Engineering, p.267-276, May 17-21, 1993, Baltimore, Maryland, United States
• 174: TAKAHASHI, M. AND KAMAYACHI, Y. 1985. An empirical study of a model for program error prediction. IEEE, 330-336.
• 175: THAYER, R., LIPOW, M., AND NELSON, E. 1978. Software Reliability. North-Holland.
• 176: W. T. Tsai , Dmitry Volovik , Thomas F. Keefe, Automated Test Case Generation for Programs Specified by Relational Algebra Queries, IEEE Transactions on Software Engineering, v.16 n.3, p.316-324, March 1990 [doi>10.1109/32.48939]
• 177: M. Z. Tsoukalas , J. W. Duran , S. C. Ntafos, On some reliability estimation problems in random and partition testing, IEEE Transactions on Software Engineering, v.19 n.7, p.687-697, July 1993 [doi>10.1109/32.238569]
• 178: Hasan Ural , B. Yang, A structural test selection criterion, Information Processing Letters, v.28 n.3, p.157-163, July 4, 1988 [doi>10.1016/0020-0190(88)90162-7]
• 179: Hasan Ural , Bo Yang, Modeling software for accurate data flow representation, Proceedings of the 15th international conference on Software Engineering, p.277-286, May 17-21, 1993, Baltimore, Maryland, United States
• 180: L. G. Valiant, A theory of the learnable, Communications of the ACM, v.27 n.11, p.1134-1142, Nov. 1984 [doi>10.1145/1968.1972]
• 181: Jeffrey Voas , Larry Morrel , Keith Miller, Predicting Where Faults Can Hide from Testing, IEEE Software, v.8 n.2, p.41-48, March 1991 [doi>10.1109/52.73748]
• 182: WEISER, M. D., GANNON, J. D., AND MCMULLIN, P.R. 1985. Comparison of structural test coverage metrics. IEEE Softw. (March), 80-85.
• 183: S. N. Weiss , E. J. Weyuker, An Extended Domain-Based Model of Software Reliability, IEEE Transactions on Software Engineering, v.14 n.10, p.1512-1524, October 1988 [doi>10.1109/32.6196]
• 184: WEYUKER, E.J. 1979a. The applicability of program schema results to programs. Int. J. Comput. Inf. Sci. 8, 5, 387-403.
• 185: WEYUKER, E.J. 1979b. Translatability and decidability questions for restricted classes of program schema. SIAM J. Comput. 8, 5, 587- 598.
• 186: WEYUKER, E.J. 1982. On testing non-testable programs. Comput. J. 25, 4, 465-470.
• 187: Elaine J. Weyuker, Assessing Test Data Adequacy through Program Inference, ACM Transactions on Programming Languages and Systems (TOPLAS), v.5 n.4, p.641-655, Oct. 1983 [doi>10.1145/69575.357231]
• 188: E J Weyuker, Axiomatizing software test data adequacy, IEEE Transactions on Software Engineering, v.12 n.12, p.1128-1138, Dec.1986
• 189: E. J. Weyuker, The evaluation of program-based software test data adequacy criteria, Communications of the ACM, v.31 n.6, p.668-675, June 1988 [doi>10.1145/62959.62963]
• 190: E. J. Weyuker, Evaluating Software Complexity Measures, IEEE Transactions on Software Engineering, v.14 n.9, p.1357-1365, September 1988 [doi>10.1109/32.6178]
• 191: WEYUKER, E. J. 1988c. An empirical study of the complexity of data flow testing. In Proceedings of SIGSOFT Symposium on Software Testing, Analysis, and Verification 2 (July), 188-195.
• 192: E. J. Weyuker, More Experience with Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.9, p.912-919, September 1993 [doi>10.1109/32.241773]
• 193: WEYUKER, E. J. AND DAVIS, M. 1983. A formal notion of program-based test data adequacy. Inf Cont. 56, 52-71.
• 194: Elaine J. Weyuker , Bingchiang Jeng, Analyzing Partition Testing Strategies, IEEE Transactions on Software Engineering, v.17 n.7, p.703-711, July 1991 [doi>10.1109/32.83906]
• 195: WEYUKER, E. J. AND OSTRAND, T.J. 1980. Theories of program testing and the application of revealing sub-domains. IEEE Trans. Softw. Eng. SE-6, 3 (May), 236-246.
• 196: WHITE, L.J. 1981. Basic mathematical definitions and results in testing. In Computer Program Testing, B. Chandrasekaran and S. Radicchi, Eds., North-Holland, 13-24.
• 197: WHITE, L. J. AND COHEN, E.I. 1980. A domain strategy for computer program testing. IEEE Trans. Softw. Eng. SE-6, 3 (May), 247-257.
• 198: Lee J. White , Bogdan Wiszniewski, Path testing of computer programs with loops using a tool for simple loop patterns, Software—Practice & Experience, v.21 n.10, p.1075-1102, Oct. 1991 [doi>10.1002/spe.4380211007]
• 199: WICHMANN, B.A. 1993. Why are there no measurement standards for software testing? Comput. Stand. Interfaces 15, 4, 361-364.
• 200: WICHMANN, B. A. AND COX, M. G. 1992. Problems and strategies for software component testing standards. J. Softw. Test. Ver. Rel. 2, 167-185.
• 201: WILD, C., ZEIL, S., CHEN, J., AND FENG, G. 1992. Employing accumulated knowledge to refine test cases. J. Softw. Test. Ver. Rel. 2, 2 (July), 53-68.
• 202: WISZNIEWSKI, B.W. 1985. Can domain testing overcome loop analysis? IEEE, 304-309.
• 203: WOODWARD, M. R. 1991. Concerning ordered mutation testing of relational operators. J. Softw. Test. Ver. Rel. 1, 3 (Dec.), 35-40.
• 204: WOODWARD, M. R. 1993. Errors in algebraic specifications and an experimental mutation testing tool. Softw. Eng. J. (July), 211-224.
• 205: WOODWARD, M. R. AND HALEWOOD, K. 1988. From weak to strong--dead or alive? An analysis of some mutation testing issues. In Proceedings of Second Workshop on Software Testing, Verification and Analysis (July) 152- 158.
• 206: WOODWARD, M. R., HEDLEY, D., AND HENNEL, M.A. 1980. Experience with path analysis and testing of programs. IEEE Trans. Softw. Eng. SE-6, 5 (May), 278-286.
• 207: WOODWARD, M. R., HENNEL, M. A., AND HEDLEY, D. 1980. A limited mutation approach to program testing. Tech. Rep. Dept. of Computational Science, University of Liverpool.
• 208: M. Young , R. M. Taylor, Combining Static Concurrency Analysis with Symbolic Execution, IEEE Transactions on Software Engineering, v.14 n.10, p.1499-1511, October 1988 [doi>10.1109/32.6195]
• 209: ZEIL, S. J. 1983. Testing for perturbations of program statements. IEEE Trans. Softw. Eng. SE-9, 3, (May), 335-346.
• 210: Steven J. Zeil, Perturbation testing for computation errors, Proceedings of the 7th international conference on Software engineering, p.257-265, March 26-29, 1984, Orlando, Florida, United States
• 211: Steven J. Zeil, Perturbation Techniques for Detecting Domain Errors, IEEE Transactions on Software Engineering, v.15 n.6, p.737-746, June 1989 [doi>10.1109/32.24727]
• 212: Steven J. Zeil , Faten H. Afifi , Lee J. White, Detection of linear errors via domain testing, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.4, p.422-451, Oct. 1992 [doi>10.1145/136586.136590]
• 213: ZHU, H. 1995a. Axiomatic assessment of control flow based software test adequacy criteria. Softw. Eng. J. (Sept.), 194-204.
• 214: ZHU, H. 1995b. An induction theory of software testing. Sci. China 38 (Supp.) (Sept.), 58-72.
• 215: Hong Zhu, A Formal Analysis of the Subsume Relation Between Software Test Adequacy Criteria, IEEE Transactions on Software Engineering, v.22 n.4, p.248-255, April 1996 [doi>10.1109/32.491648]
• 216: ZHU, H. 1996b. A formal interpretation of software testing as inductive inference. J. Softw. Test. Ver. Rel. 6 (July), 3-31.
• 217: ZHU, H. AND HALL, P. A. V. 1992a. Test data adequacy with respect to specifications and related properties. Tech. Rep. 92/06, Department of Computing, The Open University, UK, Jan.
• 218: ZHU, H. AND HALL, P. A.V. 1992b. Testability of programs: Properties of programs related to test data adequacy criteria. Tech. Rep. 92/05, Department of Computing, The Open University, UK, Jan.
• 219: Hong Zhu , P. A. V. Hall, Test data adequacy measurement, Software Engineering Journal, v.8 n.1, p.21-29, Jan. 1992
• 220: ZHU, H., HALL, P. A. V., AND MAY, J. 1992. Inductive inference and software testing. J. Softw. Test. Ver. Rel. 2, 2 (July), 69-82.
• 221: ZHU, H., HALL, P. A. V., AND MAY, J. 1994. Understanding software test adequacy--an axiomatic and measurement approach. In Mathematics of Dependable Systems, Proceedings of IMA First Conference (Sept., London), Oxford University Press, Oxford.
• 222: S. H. Z. Zweben , J. S. Gourlay, Comments on 'Axiomatizing Software Test Data Adequacy' by E.J. Weyuker, IEEE Transactions on Software Engineering, v.15 n.4, p.496-500, April 1989 [doi>10.1109/32.16609]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Measurement, Performance, Reliability, Verification

Keywords:
comparing testing effectiveness, fault detection, software unit test, test adequacy criteria, test coverage, testing methods