Focused iterative testing: a test automation case study

ABSTRACT

Timing-related defects are among the most difficult types of defects to catch while testing software. They are by definition difficult to reproduce and hence difficult to debug. Not all components of a software system have timing-related defects. For example, either a parser can analyze an input or it cannot. However, systems that have concurrent threads such as database systems are prone to timing-related defects. As a result, software developers must tailor testing to exploit vulnerabilities that occur because of threading. This paper presents the Focused Iterative Testing (FIT) approach, which uses a repetitive and iterative approach to find timing-related defects and target product areas with multithreaded characteristics by executing system tests with a multi-user test suite.

REFERENCES

• 1: Hardik Bati , Leo Giakoumakis , Steve Herbert , Aleksandras Surna, A genetic approach for random testing of database systems, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 2: J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering, SE-10(4):438--443, July 1984.
• 3: R. Hamlet. Random testing. Encyclopedia of Software Engineering. Wiley, 1994.
• 4: IBM. DB2 for Linux, UNIX, and Windows, Current May 2008. http://publib.boulder.ibm.com/infocenter/db2luw/v9.
• 5: Koushik Sen, Effective random testing of concurrent programs, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA [doi>10.1145/1321631.1321679]
• 6: Madanlal Musuvathi , Shaz Qadeer, Iterative context bounding for systematic testing of multithreaded programs, Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation, June 10-13, 2007, San Diego, California, USA
• 7: Vadim Mutilin, Concurrent Testing of Java Components Using Java PathFinder, Proceedings of the Second International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (isola 2006), p.53-59, November 15-19, 2006 [doi>10.1109/ISoLA.2006.53]
• 8: Kuo-Chung Tai , Richard H. Carver, Use of Sequencing Constraints for Specifying, Testing, and Debugging Concurrent Programs, Proceedings of the 1994 International Conference on Parallel and Distributed Systems, p.280-287, December 19-21, 1994
• 9: Willem Visser , Klaus Havelund , Guillaume Brat , SeungJoon Park, Model Checking Programs, Proceedings of the 15th IEEE international conference on Automated software engineering, p.3, September 11-15, 2000

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Concurrency

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Distributed databases

General Terms:
Reliability, Verification

Keywords:
database management systems, multi-threaded, testing