Unit testing concurrent software

ABSTRACT

There are many difficulties associated with developing correct multithreaded software, and many of the activities that are simple for single threaded software are exceptionally hard for multithreaded software. One such example is constructing unit tests involving multiple threads. Given, for example, a blocking queue implementation, writing a test case to show that it blocks and unblocks appropriately using existing testing frameworks is exceptionally hard. In this paper, we describe the MultithreadedTC framework which allows the construction of deterministic and repeatable unit tests for concurrent abstractions. This framework is not designed to test for synchronization errors that lead to rare probabilistic faults under concurrent stress. Rather, this framework allows us to demonstrate that code does provide specific concurrent functionality (e.g., a thread attempting to acquire a lock is blocked if another thread has the lock).

We describe the framework and provide empirical comparisons against hand-coded tests designed for Sun's Java concurrency utilities library and against previous frameworks that addressed this same issue. The source code for this framework is available under an open source license.

REFERENCES

• 1: Jsr 166: Concurrency utilities. http://www.jcp.org/en/jsr/detail?id=166, 2004.
• 2: Junit testing framework. http://www.junit.org, 2007.
• 3: Multithreadedtc. http://code.google.com/p/multithreadedtc/, 2007.
• 4: Swat4j. http://www.codeswat.com, 2007.
• 5: M. Albrecht. Using multi-threaded tests. http://groboutils.sourceforge.net/testingjunit/ using mtt.html, September 2004.
• 6: C. Beust and A. Popescu. Testng: Testing, the next generation. http://www.testng.org, 2007.
• 7: Richard H. Carver , Kuo-Chung Tai, Replay and Testing for Concurrent Programs, IEEE Software, v.8 n.2, p.66-74, March 1991 [doi>10.1109/52.73751]
• 8: Jong-Deok Choi , Harini Srinivasan, Deterministic replay of Java multithreaded applications, Proceedings of the SIGMETRICS symposium on Parallel and distributed tools, p.48-59, August 03-04, 1998, Welches, Oregon, United States [doi>10.1145/281035.281041]
• 9: O. Edelstein, E. Farchi, Y. Nir, G. Ratsaby, and S. Ur. Multithreaded java program test generation. IBM Systems Journal, 41(1):111--125, 2002.
• 10: P. B. Hansen. Reproducible testing of monitor. Softw., Pract. Exper., 8(6):721--729, 1978.
• 11: B. Long. Testing Concurrent Java Components. PhD thesis, The University of Queensland, July 2005.
• 12: B. Long, D. Hoffman, and P. Strooper. Tool support for testing concurrent java components. IEEE Transactions on Software Engineering, 29(6), 2003.
• 13: Luke Wildman , Brad Long , Paul Strooper, Testing Java Interrupts and Timed Waits, Proceedings of the 11th Asia-Pacific Software Engineering Conference, p.438-447, November 30-December 03, 2004 [doi>10.1109/APSEC.2004.97]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Experimentation, Reliability, Verification

Keywords:
JUnit test cases, concurrent abstraction, java, multithreadedTC, testing framework