A survey of coverage based testing tools

ABSTRACT

Test coverage is sometimes used as a way to measure how thoroughly software is tested. Coverage is used by software developers and sometimes by vendors to indicate their confidence in the readiness of their software. This survey studies and compares 17 coverage-based testing tools focusing on, but not restricted to coverage measurement. We also survey additional features, including program prioritization for testing, assistance in debugging, automatic generation of test cases, and customization of test reports. Such features make tools more useful and practical, especially for large-scale, real-life commercial software applications. Our initial motivations were both to understand the available test coverage tools and to compare them to a tool that we have developed, called eXVantage¹ (a tool suite that includes code coverage testing, debugging, performance profiling, and reporting). Our study shows that each tool has its unique features tailored to its application domains. Therefore this study can be used to pick the right coverage testing tools depending on various requirements.

REFERENCES

• 1: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 2: Yang, M. C. K.; Chao, A. Reliability-estimation and stopping-rules for software testing, based on repeated appearances of bugs; IEEE Transactions on Reliability, vol.44, no.2, p. 315--21, 1995.
• 3: Agitar (http://www.agitar.com/)
• 4: Avaya eXvantag (http://www.research.avayalabs.com/user/jjli/eXVantage.htm)
• 5: BullseyeCoverage (http://www.bullseye.com/index.html)
• 6: Clover (http://www.cenqua.com/clover/)
• 7: Cobertura (http://cobertura.sourceforge.net/)
• 8: CodeTest (http://www.metrowerks.com/MW/Develop/AMC/CodeTEST/default.htm)
• 9: Dynamic, DMS (http://dynamic-memory.com/)
• 10: gcov (http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html)
• 11: Parasoft Insure ++ (http://www.parasoft.com/jsp/products/home.jspproduct=Insure&itemId=63)
• 12: Intel code coverage tool(http://www.intel.com/cd/software/products/asmo-na/eng/219794.htm)
• 13: JCover (http://www.mmsindia.com/JCover.html)
• 14: Koalog (http://www.koalog.com/php/kover.php)
• 15: Parasoft Jtest (http://parasoft.com/jsp/home.jsp)
• 16: Purify Plus (http://www-306.ibm.com/software/awdtools/purifyplus/)
• 17: Semantic Designs (http://www.semdesigns.com/index.html)
• 18: TCAT (http://www.evalidator.org/TestWorks/)
• 19: Hiralal Agrawal, Dominators, super blocks, and program coverage, Proceedings of the 21st ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.25-34, January 16-19, 1994, Portland, Oregon, United States [doi>10.1145/174675.175935]
• 20: J. Jenny Li, Prioritize Code for Testing to Improve Code Coverage of Complex Software, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, p.75-84, November 08-11, 2005 [doi>10.1109/ISSRE.2005.33]
• 21: EMMA (http://emma.sourceforge.net/)
• 22: I. D. Baxter, Branch Coverage for Arbitrary Languages Made Easy(http://www.semanticdesigns.com/Company/Publications/TestCoverage.pdf)
• 23: W. Eric Wong , Jenny Li, An Integrated Solution for Testing and Analyzing Java Applications in an Industrial Setting, Proceedings of the 12th Asia-Pacific Software Engineering Conference (APSEC'05), p.576-583, December 15-17, 2005 [doi>10.1109/APSEC.2005.39]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Keywords:
automate test case generation, code coverage, coverage-based testing tool, dominator analysis, eXVantage, prioritization