Software error analysis: a real case study involving real faults and mutations

ABSTRACT

The paper reports on a first experimental comparison of software errors generated by real faults and by 1st-order mutations. The experiments were conducted on a program developed by a student from the industrial specification of a critical software from the civil nuclear field. Emphasis was put on the analysis of errors produced upon activation of 12 real faults by focusing on the mechanisms of error creation, masking, and propagation up to failure occurrence, and on the comparison of these errors with those created by 24 mutations. The results involve a total of 3730 errors recorded from program execution traces: 1458 errors were produced by the real faults, and the 2272 others by the mutations. They are in favor of a suitable consistency between errors generated by mutations and by real faults: 85% of the 2272 errors due to the mutations were also produced by the real faults. Moreover, it was observed that although the studied mutations were simple faults, they can create erroneous behaviors as complex as those identified for the real faults. This lends support to the representativeness of errors due to mutations.

REFERENCES

• Bis 89: Bishop P.G. and Pullen F.D., "Error Masking: a Source of Failure Dependency in Multi-Version Programs", Proc./st IFIP Working Conference on Dependable Computing for Critical Applications (DCCA-1), Santa-Barbara, USA, pp. 53-73, 1989.
• DeM 78: DeMillo R.A., Lipton R.J. and Sayward F.G., "Hints on Test Data Selection: Help for the Practicing Programmer", Computer, vol. 11, no. 4, pp. 34-41, 1978.
• DeM 94: DeMillo R.A. and Mathur A.P., "On the Use of Software Artifacts to Evaluate the Effectiveness of Mutation Analysis for Detecting Errors in Production Software", Software Engineering Research Center report, Purdue University, W. Lafayette, USA, 1994.
• Gor 93: Tarak Goradia, Dynamic impact analysis: a cost-effective technique to enforce error-propagation, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.171-181, June 28-30, 1993, Cambridge, Massachusetts, United States
• Lap 92: J.C. C. Laprie , A. Avizienis , H. Kopetz, Dependability: Basic Concepts and Terminology, Springer-Verlag New York, Inc., Secaucus, NJ, 1992
• Mur 94: Branson W. Murrill , Larry Morell, An experimental approach to analyzing software semantics using error flow information, Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis, p.200, August 17-19, 1994, Seattle, Washington, United States [doi>10.1145/186258.187200]
• Off 89: A. Offutt, The coupling effect: fact or fiction, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.131-140, December 13-15, 1989, Key West, Florida, United States
• Pod 90: A. Podgurski , L. A. Clarke, A Formal Model of Program Dependences and its Implications for Software Testing, Debugging, and Maintenance, IEEE Transactions on Software Engineering, v.16 n.9, p.965-979, September 1990 [doi>10.1109/32.58784]
• Ric 88: Richardson D.j. and Thompson M.C., "The RELAY Model of Error Detection and its Application", Proc. 2nd Workshop on Software Testing, Verification and Analysis, Banff, Canada, pp. 223-230, 1988.
• Ric 93: Debra J. Richardson , Margaret C. Thompson, An Analysis of Test Data Selection Criteria Using the RELAY Model of Fault Detection, IEEE Transactions on Software Engineering, v.19 n.6, p.533-553, June 1993 [doi>10.1109/32.232020]
• The 93: P. Thévenod-Fosse , H. Waeselynck, STATEMATE applied to statistical software testing, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.99-109, June 28-30, 1993, Cambridge, Massachusetts, United States
• The 95: Th6venod-Fosse P. and Crouzet Y., "On the Adequacy of Functional Test Criteria Based on Software Behaviour Models", Proc. 5th IFIP Working Conference on Dependable Computing for Critical Applications (DCCA-5), Urbana- Champaign, USA, pp. 176-187, 1995.
• Tho 93: Margaret C. Thompson , Debra J. Richardson , Lori A. Clarke, An information flow model of fault detection, Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, p.182-192, June 28-30, 1993, Cambridge, Massachusetts, United States
• Voa 92a: Voas J.M. and Miller K.W., "The Revealing Power of a Test Case", Journal of Software Testing, Verification and Reliability, vol. 2, pp. 25-42, 1992.
• Voa 92b: Jeffrey Voas , Larry Morrel , Keith Miller, Predicting Where Faults Can Hide from Testing, IEEE Software, v.8 n.2, p.41-48, March 1991 [doi>10.1109/52.73748]
• Zei 89: Steven J. Zeil, Perturbation Techniques for Detecting Domain Errors, IEEE Transactions on Software Engineering, v.15 n.6, p.737-746, June 1989 [doi>10.1109/32.24727]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Error handling and recovery

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Experimentation, Measurement, Verification