One evaluation of model-based testing and its automation

ABSTRACT

Model-based testing relies on behavior models for the generation of model traces: input and expected output---test cases---for an implementation. We use the case study of an automotive network controller to assess different test suites in terms of error detection, model coverage, and implementation coverage. Some of these suites were generated automatically with and without models, purely at random, and with dedicated functional test selection criteria. Other suites were derived manually, with and without the model at hand. Both automatically and manually derived model-based test suites detected significantly more requirements errors than hand-crafted test suites that were directly derived from the requirements. The number of detected programming errors did not depend on the use of models. Automatically generated model-based test suites detected as many errors as hand-crafted model-based suites with the same number of tests. A sixfold increase in the number of model-based tests led to an 11% increase in detected errors.

REFERENCES

• 1: P. Ammann and P. Black. Abstracting Formal Specifications to Generate Software Tests via Model Checking. In Proc. 18th Digital Avionics Systems Conference (DASC'99), volume 2, pages 10.A.6.1--10, October 1999.
• 2: A. Baresel, M. Conrad, S. Sadeghipour, and J. Wegener. The Interplay between Model Coverage and Code Coverage. In Proc. EuroCAST, December 2003.
• 3: Barry W. Boehm , Terence E. Gray , Thomas Seewaldt, Prototyping versus specifying: a multiproject experiment, IEEE Transactions on Software Engineering, v.10 n.3, p.290-302, May 1984 [doi>10.1109/TSE.1984.5010238]
• 4: B. Boehm, R. McClean, and D. Urfrig. Some Experience with Automated Aids to the Design of Large-Scale Reliable Software. IEEE TSE, SE-1(1):125--133, 1975.
• 5: J. Duran and S. Ntafos. An Evaluation of Random Testing. IEEE TSE, SE-10(4):438--444, July 1984.
• 6: Michael Fagan, A history of software inspections, Software pioneers: contributions to software engineering, Springer-Verlag New York, Inc., New York, NY, 2002
• 7: Phyllis G. Frankl , Oleg Iakounenko, Further empirical studies of test effectiveness, Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering, p.153-162, November 01-05, 1998, Lake Buena Vista, Florida, United States
• 8: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 9: M. Girgis and M. Woodward. An experimental comparison of the error exposing ability of program testing criteria. In Proc. IEEE/ACM workshop on software testing, pages 64--73, July 1986.
• 10: Walter J. Gutjahr, Partition Testing vs. Random Testing: The Influence of Uncertainty, IEEE Transactions on Software Engineering, v.25 n.5, p.661-674, September 1999 [doi>10.1109/32.815325]
• 11: G. Hahn , J. Philipps , A. Pretschner , T. Stauner, Prototype-Based Tests for Hybrid Reactive Systems, Proceedings of the 14th IEEE International Workshop on Rapid System Prototyping (RSP'03), p.78, June 09-11, 2003
• 12: Dick Hamlet , Ross Taylor, Partition Testing Does Not Inspire Confidence (Program Testing), IEEE Transactions on Software Engineering, v.16 n.12, p.1402-1411, December 1990 [doi>10.1109/32.62448]
• 13: R. Hamlet, Theoretical comparison of testing methods, Proceedings of the ACM SIGSOFT '89 third symposium on Software testing, analysis, and verification, p.28-37, December 13-15, 1989, Key West, Florida, United States
• 14: Mats P. E. Heimdahl , Devaraj George, Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing, Proceedings of the 19th IEEE international conference on Automated software engineering, p.176-185, September 20-24, 2004 [doi>10.1109/ASE.2004.67]
• 15: M. Heimdahl, D. George, and R. Weber. Specification Test Coverage Adequacy Criteria = Specification Test Generation Inadequacy Criteria? In Proc. 8th IEEE High Assurance in Systems Engineering Workshop, February 2004.
• 16: Hyoung Seok Hong , Insup Lee , Oleg Sokolsky , Hasan Ural, A Temporal Logic Based Theory of Test Coverage and Generation, Proceedings of the 8th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, p.327-341, April 08-12, 2002
• 17: Franz Huber , Bernhard Schätz , Geralf Einert, Consistent Graphical Specification of Distributed Systems, Proceedings of the 4th International Symposium of Formal Methods Europe on Industrial Applications and Strengthened Foundations of Formal Methods, p.122-141, September 15-19, 1997
• 18: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 19: MOST Cooperation. MOST Media Oriented System Transport---Multimedia and Control Networking Technology. MOST Specification Rev 2.2, Ver 2.2-00. 2002.
• 20: John D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper, Authorhouse, 2004
• 21: Simeon C. Ntafos, An evaluation of required element testing strategies, Proceedings of the 7th international conference on Software engineering, p.250-256, March 26-29, 1984, Orlando, Florida, United States
• 22: A Jefferson Offutt , Yiwei Xiong , Shaoying Liu, Criteria for Generating Specification-Based Tests, Proceedings of the 5th International Conference on Engineering of Complex Computer Systems, p.119, October 18-22, 1999
• 23: J. Philipps, A. Pretschner, O. Slotosch, E. Aiglstorfer, S. Kriebel, and K. Scholl. Model-based test case generation for smart cards. In Proc. 8th Intl. Workshop on Formal Methods for Industrial Critical Syst., pages 168--192, 2003.
• 24: A. Pretschner. Compositional generation of MC/DC test suites. ENTCS, 82(6):1--11, 2003.
• 25: A. Pretschner , O. Slotosch , E. Aiglstorfer , S. Kriebel, Model-based testing for real: The inhouse card case study, International Journal on Software Tools for Technology Transfer (STTT), v.5 n.2, p.140-157, March 2004 [doi>10.1007/s10009-003-0128-3]
• 26: E. Weyuker , T. Goradia , A. Singh, Automatically Generating Test Data from a Boolean Specification, IEEE Transactions on Software Engineering, v.20 n.5, p.353-363, May 1994 [doi>10.1109/32.286420]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
      D.2.2 Design Tools and Techniques

General Terms:
Verification

Keywords:
CASE, abstraction, automotive software, coverage, model-based development, test case generation