TestFul: using a hybrid evolutionary algorithm for testing stateful systems

ABSTRACT

This paper introduces TestFul, a framework for testing stateful systems and focuses on object-oriented software. TestFul employs a hybrid multi-objective evolutionary algorithm, to explore the space of feasible tests efficiently, and novel quality metrics, based on both def-use pairs and behavioral coverage, to judge the quality of tests.

We compare our framework against random testing by considering the level of coverage, the size of generated tests, and the time required to generate the tests. Our preliminary results show the validity of the approach: TestFul outperforms random testing in most of the cases.

REFERENCES

• 1: Ilinca Ciupa , Andreas Leitner , Manuel Oriol , Bertrand Meyer, Experimental assessment of random testing for object-oriented software, Proceedings of the 2007 international symposium on Software testing and analysis, July 09-12, 2007, London, United Kingdom [doi>10.1145/1273463.1273476]
• 2: Valentin Dallmeier , Christian Lindig , Andrzej Wasylkowski , Andreas Zeller, Mining object behavior with ADABU, Proceedings of the 2006 international workshop on Dynamic systems analysis, May 23-23, 2006, Shanghai, China [doi>10.1145/1138912.1138918]
• 3: R. Hamlet. Random testing. In Encyclopedia of Software Engineering, pages 970--978. Wiley, 1994.
• 4: B. Korel, Automated Software Test Data Generation, IEEE Transactions on Software Engineering, v.16 n.8, p.870-879, August 1990 [doi>10.1109/32.57624]
• 5: Andreas Leitner , Ilinca Ciupa , Manuel Oriol , Bertrand Meyer , Arno Fiva, Contract driven development = test driven development - writing test cases, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287685]
• 6: Rupak Majumdar , Koushik Sen, Hybrid Concolic Testing, Proceedings of the 29th international conference on Software Engineering, p.416-426, May 20-26, 2007 [doi>10.1109/ICSE.2007.41]
• 7: Phil McMinn, Search-based software test data generation: a survey: Research Articles, Software Testing, Verification & Reliability, v.14 n.2, p.105-156, June 2004 [doi>10.1002/stvr.v14:2]
• 8: Paolo Tonella, Evolutionary testing of classes, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 9: Vim. http://www.vim.org.
• 10: A. Watkins. The automatic generation of test data using genetic algorithms. In Proceedings of the Fourth Software Quality Converence, pages 300--309, 1995.
• 11: E. Zitzler, M. Laumanns, and L. Thiele. SPEA2: Improving the Strength Pareto Evolutionary Algorithm. Technical report, ETH Zurich, May 2001.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Algorithms

Keywords:
evolutionary algorithms, multi-objective optimization, object-oriented paradigm, stateful systems, test generation