Software performance testing using covering arrays: efficient screening designs with categorical factors

ABSTRACT

Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis, while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.

REFERENCES

• 1: Alphaworks:IBM. Combinatorial test services tool. http://www.alphaworks.ibm.com/tech/cts.
• 2: David M. Cohen , Siddhartha R. Dalal , Michael L. Fredman , Gardner C. Patton, The AETG System: An Approach to Testing Based on Combinatorial Design, IEEE Transactions on Software Engineering, v.23 n.7, p.437-444, July 1997 [doi>10.1109/32.605761]
• 3: A. Hartman and L. Raskin. Problems and algorithms for covering arrays. Discrete Math, 284(3):149--156, Dec 2003.
• 4: Dean Hoskins , Renée C. Turban , Charles J. Colbourn, Experimental designs in software engineering: d-optimal designs and covering arrays, Proceedings of the 2004 ACM workshop on Interdisciplinary software engineering research, November 05-05, 2004, Newport Beach, CA, USA [doi>10.1145/1029997.1030009]
• 5: R. Meyer and C. Nachtsheim. The coordinate-exchange algorithm for constructing exact optimal experimental designs. Technometrics, 37(1):60--69, Feb 1995.
• 6: Toby J. Mitchell, An algorithm for the construction of “D-optimal” experimental designs, Technometrics, v.42 n.1, p.48-54, Feb. 2000 [doi>10.2307/1271431]
• 7: D. C. Montgomery. Design and Analysis of Experiments (Fifth Edition). John Wiley and Sons, New York NY, 2001.
• 8: Renée C. Bryce , Charles J. Colbourn , Myra B. Cohen, A framework of greedy methods for constructing interaction test suites, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062495]
• 9: K. Vadde and V. R. Syrotiuk. Factor interaction on service delivery in mobile ad hoc networks. IEEE Journal on Selected Areas in Communications, 22(7):1335-- 1346, Sept. 2004.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)
      D.2.8 Metrics
          Subjects: Performance measures; Complexity measures

Keywords:
D-optimal designs, covering arrays, performance testing