HOTTest: A model-based test design technique for enhanced testing of domain-specific applications

ABSTRACT

Model-based testing is an effective black-box test generation technique for applications. Existing model-based testing techniques, however, fail to capture implicit domain-specific properties, as they overtly rely on software artifacts such as design documents, requirement specifications, etc., for completeness of the test model. This article presents a technique, HOTTest, which uses a strongly typed domain-specific language to model the system under test. This allows extraction of type-related system invariants, which can be related to various domain-specific properties of the application. Thus, using HOTTest, it is possible to automatically extract and embed domain-specific requirements into the test models. In this article we describe HOTTest, its principles and methodology, and how it is possible to relate domain-specific properties to specific type constraints. HOTTest is described using the example of HaskellDB, which is a Haskell-based embedded domain-specific language for relational databases. We present an example application of the technique and compare the results to some other commonly used Model-based test automation techniques like ASML-based testing, UML-based testing, and EFSM-based testing.

REFERENCES

• 1: Abdurazik, A. and Offutt, J. 2000. Using UML collaboration diagrams for static checking and test generation. In Proceedings of the 3rd International Conference on the Unified Modeling Language (York, UK, Oct.). 383--395.
• 2: G. Arango, Domain analysis: from art form to engineering discipline, Proceedings of the 5th international workshop on Software specification and design, p.152-159, April 1989, Pittsburgh, Pennsylvania, United States
• 3: Barnett, M., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., and Veanes, M. 2003. Model-Based testing with AsmL.NET. In Proceedings of the 1st European Conference on Model-Driven Software Engineering (Dec.).
• 4: Basili, V. 1995. Software quality assurance and measurement: A worldwide perspective. In Applying the Goal/Question/Metric Paradigm in the Experience Factory. International Thomson Computer Press, Boston, 21--44.
• 5: Grady Booch , James Rumbaugh , Ivar Jacobson, Unified Modeling Language User Guide, The (2nd Edition) (Addison-Wesley Object Technology Series), Addison-Wesley Professional, 2005
• 6: Campbell, D. T. and Stanley, J. C. 1963. Experimental and Quasi-Experimental Designs for Research. Rand McNally and Company, Chicago.
• 7: Jarir K. Chaar , Michael J. Halliday , Inderpal S. Bhandari , Ram Chillarege, In-Process Evaluation for Software Inspection and Test, IEEE Transactions on Software Engineering, v.19 n.11, p.1055-1070, November 1993 [doi>10.1109/32.256853]
• 8: David Chays , Saikat Dan , Phyllis G. Frankl , Filippos I. Vokolos , Elaine J. Weber, A framework for testing database applications, Proceedings of the 2000 ACM SIGSOFT international symposium on Software testing and analysis, p.147-157, August 21-24, 2000, Portland, Oregon, United States
• 9: Ram Chillarege , Inderpal S. Bhandari , Jarir K. Chaar , Michael J. Halliday , Diane S. Moebus , Bonnie K. Ray , Man-Yuen Wong, Orthogonal Defect Classification-A Concept for In-Process Measurements, IEEE Transactions on Software Engineering, v.18 n.11, p.943-956, November 1992 [doi>10.1109/32.177364]
• 10: Arie van Deursen , Paul Klint , Joost Visser, Domain-specific languages: an annotated bibliography, ACM SIGPLAN Notices, v.35 n.6, p.26-36, June 2000 [doi>10.1145/352029.352035]
• 11: Jeremy Dick , Alain Faivre, Automating the Generation and Sequencing of Test Cases from Model-Based Specifications, Proceedings of the First International Symposium of Formal Methods Europe on Industrial-Strength Formal Methods, p.268-284, April 19-23, 1993
• 12: R. Dssouli , K. Saleh , E. Aboulhamid , A. En-Nouaary , C. Bourhfir, Test development for communication protocols: towards automation, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.17, p.1835-1872, June 7, 1999 [doi>10.1016/S1389-1286(99)00063-8]
• 13: EMPIRIX 1999. Test Master User's Guide, Release 1.9.5. Empirix, New Hampshire.
• 14: GALOIS 2002. HaskellDB Manual. Galois Connections, Portland, OR.
• 15: Marie-Claude Gaudel, Testing Can Be Formal, Too, Proceedings of the 6th International Joint Conference CAAP/FASE on Theory and Practice of Software Development, p.82-96, May 22-26, 1995
• 16: P. A. V. Hall, Relationship between specifications and testing, Information and Software Technology, v.33 n.1, p.47-52, Jan./Feb. 1991 [doi>10.1016/0950-5849(91)90023-5]
• 17: Hartman, A. and Nagin, K. 2005. The AGEDIS tools for model-based testing. LNCS 3297, Springer-Verlag, New York, 277--280.
• 18: Paul Hudak, The Haskell school of expression: learning functional programming through multimedia, Cambridge University Press, New York, NY, 2000
• 19: IBM 2004. Rational XDE Tester User's Guide. IBM, New York.
• 20: IEEE 1984. IEEE Guide to Software Requirements Specification. IEEE, Piscataway, NJ.
• 21: Lalita Jategaonkar Jagadeesan , Adam Porter , Carlos Puchol , J. Christopher Ramming , Lawrence G. Votta, Specification-based testing of reactive software: tools and experiments: experience report, Proceedings of the 19th international conference on Software engineering, p.525-535, May 17-23, 1997, Boston, Massachusetts, United States [doi>10.1145/253228.253435]
• 22: Daan Leijen , Erik Meijer, Domain specific embedded compilers, Proceedings of the 2nd conference on Domain-specific languages, p.109-122, October 03-06, 1999, Austin, Texas, United States
• 23: Atif M. Memon , Martha E. Pollack , Mary Lou Soffa, Hierarchical GUI Test Case Generation Using Automated Planning, IEEE Transactions on Software Engineering, v.27 n.2, p.144-155, February 2001 [doi>10.1109/32.908959]
• 24: MERCURY 2001.WinRunner User's Guide, version 7.01.Mercury Interactive, Sunnyvale, CA.
• 25: Dennis K. Peters , David Lorge Parnas, Using Test Oracles Generated from Program Documentation, IEEE Transactions on Software Engineering, v.24 n.3, p.161-173, March 1998 [doi>10.1109/32.667877]
• 26: A. Pretschner , W. Prenninger , S. Wagner , C. Kühnel , M. Baumgartner , B. Sostawa , R. Zölch , T. Stauner, One evaluation of model-based testing and its automation, Proceedings of the 27th international conference on Software engineering, p.392-401, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062529]
• 27: RATIONAL 2002. Rational Rose Enterprise Edition. IBM, New York.
• 28: Arthur Alexander Reyes , Debra Richardson, Siddhartha: A Method for Developing Domain-Specific Test Driver Generators, Proceedings of the 14th IEEE international conference on Automated software engineering, p.81, October 12-15, 1999
• 29: Savage, P., Walters, S., and Stephenson, M. 1997. Automated test methodology for operational flight programs. In Proceedings of the IEEE Aerospace Conference, 293--305.
• 30: SEGUE 2002. Silktest User's Guide, version 6.5. Segue Software, Lexington, MA.
• 31: Avik Sinha , Carol S. Smidts, Domain specific test case generation using higher ordered typed languages for specification, University of Maryland at College Park, College Park, MD, 2005
• 32: Avik Sinha , Carol Smidts, An experimental evaluation of a higher-ordered-typed-functional specification-based test-generation technique, Empirical Software Engineering, v.11 n.2, p.173-202, June 2006 [doi>10.1007/s10664-006-6401-9]
• 33: Stocks, P. A. and Carrington, D. A. 1991. Deriving software test cases from formal specifications. In Proceedings of the 6th Australian Software Engineering Conference (Sydney, Australia). 327--340.
• 34: Simon Thompson, The Haskell: The Craft of Functional Programming, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 35: Tretmans, J. and Belinfante, A. 1999. Automatic testing with formal methods. In Proceedings of EuroSTAR'99: 7th European International Conference on Software Testing, Analysis and Review (Barcelona, Nov.). 8--12.
• 36: Boris Vaysburg , Luay H. Tahat , Bogdan Korel, Dependence analysis in reduction of requirement based test suites, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 37: Margus Veanes , Colin Campbell , Wolfram Schulte , Nikolai Tillmann, Online testing with model programs, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 38: Wang, C. J. and Liu, M. T. 1993. Generating test cases for EFSM with given fault models. In Proceedings of the IEEE Infocom Conference 2, 774--781.
• 39: Clay E. Williams, Towards a Test-Ready Meta-model for Use Cases, Workshop of the pUML-Group held together with the «UML»2001 on Practical UML-Based Rigorous Development Methods - Countering or Integrating the eXtremists, p.270-287, October 01, 2001

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Languages, Performance, Reliability

Keywords:
Haskell, HaskellDB, Test case generation, database-specific test case generation, domain-specific languages, domain-specific testing, model-based testing, test generation tools