Towards the determination of typical failure patterns

ABSTRACT

One possibility to make testing strategies more effective is to incorporate knowledge about the typical geometric structure of failure-causing inputs within the input domain into the test data selection. For example Adaptive Random Testing is a testing strategy which is based on the idea of failure-causing inputs being clustered within the input domain. So far, there has been no empirical quantification about the location and the geometric shape of failure-causing inputs. Thus it is currently unknown whether the encouraging results gained by Adaptive Random Testing hold true in general. This work aims at introducing an approach which makes it possible to verify the assumption of clustered failure patterns. Possibly it furthermore enables the improvement of current Adaptive Random Testing methods and the development of further black box testing strategies incorporating knowledge about location and shape of failure patterns into test data selection. Therefore metrics for location and shape of failure patterns are specified. They are based on methods from image analysis.

REFERENCES

• 1: Paul E. Ammann , John C. Knight, Data Diversity: An Approach to Software Fault Tolerance, IEEE Transactions on Computers, v.37 n.4, p.418-425, April 1988 [doi>10.1109/12.2185]
• 2: P. G. Bishop. The variation of software survival time for different operational input profiles. In Proceedings of the 23rd International Conference on Fault-Tolerant Computing (FTCS '93), pages 98--107. IEEE Computer Society Press, 1993.
• 3: F. T. Chan, T. Y. Chen, I. K. Mak, and Y. T. Yu. Proportional sampling strategy: Guidelines for software testing practitioners. Information and Software Technology, 38:775--782, 1996.
• 4: T. Chen, F.-C. Kuo, and C. Sun. Impact of the compactness of failure regions on the performance of adaptive random testing. Journal of Software, 17:2438--2449, 2006.
• 5: T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference (ASIAN '04), volume 3321 of Lecture Notes in Computer Science, pages 320--329. Springer, 2004.
• 6: G. B. Finelli. Nasa software failure characterization experiments. Reliability Engineering & System Safety, 32:155--169, 1991.
• 7: Yu-Seung Ma , Jeff Offutt , Yong Rae Kwon, MuJava: an automated class mutation system: Research Articles, Software Testing, Verification & Reliability, v.15 n.2, p.97-133, June 2005 [doi>10.1002/stvr.v15:2]
• 8: Johannes Mayer , Christoph Schneckenburger, Adaptive Random Testing with Enlarged Input Domain, Proceedings of the Sixth International Conference on Quality Software, p.251-258, October 27-28, 2006 [doi>10.1109/QSIC.2006.8]
• 9: Johannes Mayer , Christoph Schneckenburger, An empirical analysis and comparison of random testing techniques, Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering, September 21-22, 2006, Rio de Janeiro, Brazil [doi>10.1145/1159733.1159751]
• 10: R. Merkel. Analysis and Enhancements of Adaptive Random Testing. PhD thesis, Swinburne University of Technology, Australia, 2005.
• 11: A. Jefferson Offutt, Investigations of the software testing coupling effect, ACM Transactions on Software Engineering and Methodology (TOSEM), v.1 n.1, p.5-20, Jan. 1992 [doi>10.1145/125489.125473]
• 12: J. Ohser and F. Mücklich. Statistical Analysis of Microstructures in Materials Science. John Wiley & Sons, 2000.
• 13: W. H. Press, B. P. Flannery, S. A. Teulolsky, and W. T. Vetterling. Numerical Recipes. Cambridge University Press, 1986.
• 14: M. J. P. van der Meulen , P. G. Bishop , M. Revilla, An Exploration of Software Faults and Failure Behaviour in a Large Population of Programs, Proceedings of the 15th International Symposium on Software Reliability Engineering, p.101-112, November 02-05, 2004 [doi>10.1109/ISSRE.2004.7]
• 15: L. J. White , E. I. Cohen, A Domain Strategy for Computer Program Testing, IEEE Transactions on Software Engineering, v.6 n.3, p.247-257, May 1980 [doi>10.1109/TSE.1980.234486]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Verification

Keywords:
failure location, failure pattern, failure-causing input