Automated, contract-based user testing of commercial-off-the-shelf components

ABSTRACT

Commercial-off-the-Shelf (COTS) components provide a means to construct software (component-based) systems in reduced time and cost. In a COTS component software market there exist component vendors (original developers of the component) and component users (developers of the component-based systems). The former provide the component to the user without source code or design documentation, and as a result it is difficult for the latter to adequately test the component when deployed in their system. In this article we propose a framework that clarifies the roles and responsibilities of both parties so that the user can adequately test the component in a deployment environment and the vendor does not need to release proprietary details. Then, based on this framework we combine and adapt two specification-based testing techniques and describe (and implement) a method for the automated generation of adequate test sets. An evaluation of our approach on a case study demonstrates that it is possible to automatically generate cost effective test sequences and that these test sequences are effective at detecting complex errors.

REFERENCES

• 1: Paul Ammann , Jeff Offutt , Hong Huang, Coverage Criteria for Logical Expressions, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.99, November 17-21, 2003
• 2: J. H. Andrews , L. C. Briand , Y. Labiche, Is mutation an appropriate tool for testing experiments?, Proceedings of the 27th international conference on Software engineering, p.402-411, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062530]
• 3: Sami Beydeda , Volker Gruhn, An Integrated Testing Technique for Component-Based Software, Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications, p.328, June 25-29, 2001
• 4: S. Beydeda and V. Gruhn, Testing COTS Components and Systems, Springer, 2005.
• 5: Robert V. Binder, Testing object-oriented systems: models, patterns, and tools, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 6: R. G. Busacker and P. J. Gowen, "A Procedure for Determining a Family of Minimal-Cost Network Flow Patterns," John Hopkins University, O.R.O., Technical Report 15, 1961.
• 7: Richard H. Carver , Kuo-Chung Tai, Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs, IEEE Transactions on Software Engineering, v.24 n.6, p.471-490, June 1998 [doi>10.1109/32.689403]
• 8: A. Cechich and M. Polo, "COTS Component Testing through Aspect-Based Metadata," in S. Beydeda and V. Gruhn, Eds., Testing COTS Components and Systems, Springer, pp. 71--88, 2005.
• 9: N. Christofides, V. Campos, A. Corberan and E. Mota, "An algorithm for the rural postman problem on a directed graph," Mathematical Programming Studies, vol. 26, pp. 155--166, 1986.
• 10: F. J. Daniels , K. C. Tai, Measuring the Effectiveness of Method Test Sequences Derived from Sequencing Constraints, Proceedings of the Technology of Object-Oriented Languages and Systems, p.74, August 01-05, 1999
• 11: J. L. Devore, Probability and Statistics for Eng. and the Sciences, Duxbury Press, 5th Edition, 1999.
• 12: M. Dror, Arc Routing: Theory, Solutions and Applications, Kluwer, 2000.
• 13: J. Edmonds, "Optimum branchings," Journal Research of the National Bureau of Standards 71B, pp. --33-240, 1967.
• 14: Jerry Zayu Gao , Jacob Tsao , Ye Wu , Taso H.-S. Jacob, Testing and Quality Assurance for Component-Based Software, Artech House, Inc., Norwood, MA, 2003
• 15: Sudipto Ghosh , Aditya P. Mathur, Interface Mutation to Assess the Adequacy of Tests for Components and Systems, Proceedings of the Technology of Object-Oriented Languages and Systems (TOOLS 34'00), p.37, July 30-August 03, 2000
• 16: JBoss, JBoss Application Server, http://www.jboss.com (Last accessed 26 Jan. 2005).
• 17: Bengi Karaçali , Kuo-Chung Tai, Automated Test Sequence Generation Using Sequencing Constraints for Concurrent Programs, Proceedings of the International Symposium on Software Engineering for Parallel and Distributed Systems, p.97, May 17-18, 1999
• 18: S. Kim, J. A. Clark and J. A. McDermid, "Class Mutation: Mutation Testing for Object-Oriented Programs," Proc. Net.ObjectDays, 2000.
• 19: K. N. King , A. Jefferson Offutt, A Fortran language system for mutation-based software testing, Software—Practice & Experience, v.21 n.7, p.685-718, July 1991 [doi>10.1002/spe.4380210704]
• 20: Vincent Massol , Ted Husted, JUnit in Action, Manning Publications Co., Greenwich, CT, 2003
• 21: OMG, "UML 2.0 Superstructure Specification," Object Management Group, Final Adopted Specification ptc/03-08-02, 2003.
• 22: Using Component Metacontent to Support the Regression Testing of Component-Based Software, Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01), p.716, November 07-09, 2001
• 23: T. J. Ostrand , M. J. Balcer, The category-partition method for specifying and generating fuctional tests, Communications of the ACM, v.31 n.6, p.676-686, June 1988 [doi>10.1145/62959.62964]
• 24: A. Polini and A. Bertolino, "A User-Oriented Framework for Component Deployment Testing," in S. Beydeda and V. Gruhn, Eds., Testing COTS Components and Systems, Springer, pp. 169--194, 2005.
• 25: D. S. Rosenblum, "Adequate testing of component-based software," Department of Information and Computer Science, University of California, Technical Report 97--34, 1997.
• 26: Sun Microsystems, Enterprise JavaBeans Technology, java.sun.com/products/ejb/ (Last accessed 26 Jan. 2005)
• 27: Sun Microsystems, Java Blueprints, java.sun.com/ reference/blueprints/ (Last accessed 26 Jan. 2005).
• 28: Clemens Szyperski, Component software: beyond object-oriented programming, ACM Press/Addison-Wesley Publishing Co., New York, NY, 1998
• 29: Jos Warmer , Anneke Kleppe, The Object Constraint Language: Getting Your Models Ready for MDA, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2003
• 30: E. Weyuker , T. Goradia , A. Singh, Automatically Generating Test Data from a Boolean Specification, IEEE Transactions on Software Engineering, v.20 n.5, p.353-363, May 1994 [doi>10.1109/32.286420]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Validation

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

General Terms:
Algorithms, Design, Management, Reliability, Verification

Keywords:
COTS, UML, adequacy criteria, component