Faults' context matters

ABSTRACT

When choosing a testing technique, practitioners want to know which one will detect the faults that matter most to them in the programs that they plan to test. Do empirical evaluations of testing techniques provide this information? More often than not, they report how many faults in a carefully chosen "representative" sample the evaluated techniques detect. But the population of faults that such a sample would represent depends heavily on the faults' context or environment---as does the cost of failing to detect those faults. If empirical studies are to provide information that a practitioner can apply outside the context of the study, they must characterize the faults studied in a way that translates across contexts. A testing technique's fault-detecting abilities could then be interpreted relative to the fault characterization. In this paper, we present a list of criteria that a fault characterization must meet in order to be fit for this task, and we evaluate several well-known fault characterizations against the criteria. Two families of characterizations are found to satisfy the criteria: those based on graph models of programs and those based on faults' detection by testing techniques.

REFERENCES

• 1: J. H. Andrews , L. C. Briand , Y. Labiche, Is mutation an appropriate tool for testing experiments?, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062530]
• 2: Victor R. Basili , Richard W. Selby, Comparing the Effectiveness of Software Testing Strategies, IEEE Transactions on Software Engineering, v.13 n.12, p.1278-1296, December 1987 [doi>10.1109/TSE.1987.232881]
• 3: T. Dinh-Trong, S. Ghosh, R. France, B. Baudry, and F. Fleury. A taxonomy of faults for UML designs. In 2nd MoDeVa workshop - Model design and validation, 2005.
• 4: Sebastian Elbaum , Alexey Malishevsky , Gregg Rothermel, Incorporating varying test costs and fault severities into test case prioritization, Proceedings of the 23rd International Conference on Software Engineering, p.329-338, May 12-19, 2001, Toronto, Ontario, Canada
• 5: Mary Jean Harrold , A. Jefferson Offutt , Kanupriya Tewary, An approach to fault modeling and fault seeding using the program dependence graph, Journal of Systems and Software, v.36 n.3, p.273-295, March 1997 [doi>10.1016/S0164-1212(96)00175-6]
• 6: W. E. Howden, Reliability of the Path Analysis Testing Strategy, IEEE Transactions on Software Engineering, v.2 n.3, p.208-215, May 1976 [doi>10.1109/TSE.1976.233816]
• 7: A. M. Memon. Software-testing benchmarks. http://www.cs.umd.edu/~atif/Benchmarks/.
• 8: John C. Munson , Allen P. Nikora, Toward A Quantifiable Definition of Software Faults, Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE'02), p.388, November 12-15, 2002
• 9: John D. Musa, Operational Profiles in Software-Reliability Engineering, IEEE Software, v.10 n.2, p.14-32, March 1993 [doi>10.1109/52.199724]
• 10: A. Jefferson Offutt , J. Huffman Hayes, A semantic model of program faults, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, p.195-200, January 08-10, 1996, San Diego, California, United States
• 11: G. Rothermel, S. Elbaum, A. Kinneer, and H. Do. Software-artifact infrastructure repository. http://sir.unl.edu/portal/.
• 12: E. J. Weyuker. Can we measure software testing effectiveness? In Proceedings of the 1st International Software Metrics Symposium, pages 100--107, 1993.
• 13: Qing Xie , Atif M. Memon, Designing and comparing automated test oracles for GUI-based software applications, ACM Transactions on Software Engineering and Methodology (TOSEM), v.16 n.1, p.4-es, February 2007 [doi>10.1145/1189748.1189752]