Experimental assessment of random testing for object-oriented software

ABSTRACT

Progress in testing requires that we evaluate the effectiveness of testing strategies on the basis of hard experimental evidence, not just intuition or a priori arguments. Random testing, the use of randomly generated test data, is an example of a strategy that the literature often deprecates because of such preconceptions. This view is worth revisiting since random testing otherwise offers several attractive properties: simplicity of implementation, speed of execution, absence of human bias.

We performed an intensive experimental analysis of the efficiency of random testing on an existing industrial-grade code base. The use of a large-scale cluster of computers, for a total of 1500 hours of CPU time, allowed a fine-grain analysis of the individual effect of the various parameters involved in the random testing strategy, such as the choice of seed for a random number generator. The results provide insights into the effectiveness of random testing and a number of lessons for testing researchers and practitioners.

REFERENCES

• 1: The EiffelBase library. Eiffel Software Inc. http://www.eiffel.com/.
• 2: Jtest. Parasoft Corporation. http://www.parasoft.com/.
• 3: J. H. Andrews, S. Haldar, Y. Lei, and C. H. Li. Randomized unit testing: Tool support and best practices. Technical Report 663, Department of Computer Science, University of Western Ontario, January 2006.
• 4: James H. Andrews , Susmita Haldar , Yong Lei , Felix Chun Hang Li, Tool support for randomized unit testing, Proceedings of the 1st international workshop on Random testing, July 20-20, 2006, Portland, Maine [doi>10.1145/1145735.1145741]
• 5: Marat Boshernitsan , Roongko Doong , Alberto Savoia, From daikon to agitator: lessons and challenges in building a commercial tool for developer testing, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146258]
• 6: Chandrasekhar Boyapati , Sarfraz Khurshid , Darko Marinov, Korat: automated testing based on Java predicates, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 7: T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In M. J. Maher, editor, Advances in Computer Science - ASIAN 2004: Higher-Level Decision Making. 9th Asian Computing Science Conference. Proceedings. Springer-Verlag GmbH, 2004.
• 8: Tsong Yueh Chen , Robert Merkel, Quasi-random testing, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101957]
• 9: T. Y. Chen , Y. T. Yu, On the Relationship Between Partition and Random Testing, IEEE Transactions on Software Engineering, v.20 n.12, p.977-980, December 1994 [doi>10.1109/32.368132]
• 10: I. Ciupa and A. Leitner. Automatic testing based on Design by Contract. In Proceedings of Net.ObjectDays 2005 (6th Annual International Conference on Object-Oriented and Internet-based Technologies, Concepts, and Applications for a Networked World), pages 545--557, September 19-22 2005.
• 11: Ilinca Ciupa , Andreas Leitner , Manuel Oriol , Bertrand Meyer, Object distance and its application to adaptive random testing of object-oriented programs, Proceedings of the 1st international workshop on Random testing, July 20-20, 2006, Portland, Maine [doi>10.1145/1145735.1145744]
• 12: Christoph Csallner , Yannis Smaragdakis, JCrasher: an automatic robustness tester for Java, Software—Practice & Experience, v.34 n.11, p.1025-1050, September 2004 [doi>10.1002/spe.602]
• 13: Marcelo d'Amorim , Carlos Pacheco , Tao Xie , Darko Marinov , Michael D. Ernst, An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing, Proceedings of the 21st IEEE/ACM International Conference on Automated Software Engineering, p.59-68, September 18-22, 2006 [doi>10.1109/ASE.2006.13]
• 14: J. Duran and S. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering, SE-10:438--444, July 1984.
• 15: Patrice Godefroid , Nils Klarlund , Koushik Sen, DART: directed automated random testing, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 16: Philip J. Guo , Jeff H. Perkins , Stephen McCamant , Michael D. Ernst, Dynamic inference of abstract types, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146268]
• 17: Dick Hamlet , Ross Taylor, Partition Testing Does Not Inspire Confidence (Program Testing), IEEE Transactions on Software Engineering, v.16 n.12, p.1402-1411, December 1990 [doi>10.1109/32.62448]
• 18: R. Hamlet. Random testing. In J. Marciniak, editor, Encyclopedia of Software Engineering, pages 970--978. Wiley, 1994.
• 19: Andreas Leitner , Ilinca Ciupa , Bertrand Meyer , Mark Howard, Reconciling Manual and Automated Testing: The AutoTest Experience, Proceedings of the 40th Annual Hawaii International Conference on System Sciences, p.261a, January 03-06, 2007 [doi>10.1109/HICSS.2007.462]
• 20: S. Mankefors , R. Torkar , A. Boklund, New Quality Estimations in Random Testing, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.468, November 17-21, 2003
• 21: Bertrand Meyer, Object-oriented software construction (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1997
• 22: C. Oriat. Jartege: a tool for random generation of unit tests for Java classes. Technical Report RR-1069-I, Centre National de la Recherche Scientifique, Institut National Polytechnique de Grenoble, Universite Joseph Fourier Grenoble I, June 2004.
• 23: C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classification of test inputs. In ECOOP 2005 - Object-Oriented Programming, 19th European Conference, Glasgow, Scotland, July 25-29, 2005.
• 24: Elaine J. Weyuker , Bingchiang Jeng, Analyzing Partition Testing Strategies, IEEE Transactions on Software Engineering, v.17 n.7, p.703-711, July 1991 [doi>10.1109/32.83906]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.1 Specifying and Verifying and Reasoning about Programs

Keywords:
experimental evaluation, random testing, software testing