Advanced code coverage analysis using substring holes

ABSTRACT

Code coverage is a common aid in the testing process. It is generally used for marking the source code segments that were executed and, more importantly, those that were not executed.

Many code coverage tools exist, supporting a variety of languages and operating systems. Unfortunately, these tools provide little or no assistance when code coverage data is voluminous. Such quantities are typical of system tests and even for earlier testing phases. Drill-down capabilities that look at different granularities of the data, starting with directories and going through files to functions and lines of source code, are insufficient. Such capabilities make the assumption that the coverage issues themselves follow the code hierarchy. We argue that this is not the case for much of the uncovered code. Two notable examples are error handling code and platform-specific constructs. Both tend to be spread throughout the source in many files, even though the related coverage, or lack thereof, is highly dependent.

To make the task more manageable, and therefore more likely to be performed by users, we developed a hole analysis algorithm and tool that is based on common substrings in the names of functions. We tested its effectiveness using two large IBM software systems. In both of them, we asked domain experts to judge the results of several hole-ranking heuristics. They found that 57% - 87% of the 30 top-ranked holes identified by the effective heuristics are relevant. Moreover, these holes are often unexpected. This is especially impressive because substring hole analysis relies only on the names of functions, whereas domain experts have a broad and deep understanding of the system.

We grounded our results in a theoretical framework that states desirable mathematical properties of hole ranking heuristics. The empirical results show that heuristics with these properties tend to perform better, and do so more consistently, than heuristics lacking them.

REFERENCES

• 1: Y. Adler, E. Farchi, M. Klausner, D. Pelleg, O. Raz, M. Shochat, S. Ur, and A. Zlotnick. Automated substring hole analysis. In ICSE, NIER Track, 2009.
• 2: Rakesh Agrawal , Ramakrishnan Srikant, Fast Algorithms for Mining Association Rules in Large Databases, Proceedings of the 20th International Conference on Very Large Data Bases, p.487-499, September 12-15, 1994
• 3: Hezi Azatchi , Laurent Fournier , Eitan Marcus , Shmuel Ur , Avi Ziv , Keren Zohar, Advanced Analysis Techniques for Cross-Product Coverage, IEEE Transactions on Computers, v.55 n.11, p.1367-1379, November 2006 [doi>10.1109/TC.2006.173]
• 4: Cynthia Dwork , Ravi Kumar , Moni Naor , D. Sivakumar, Rank aggregation methods for the Web, Proceedings of the 10th international conference on World Wide Web, p.613-622, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372165]
• 5: Focus code and functional coverage tool. http://www.alphaworks.ibm.com/tech/focus. Accessed December 2008.
• 6: Raanan Grinwald , Eran Harel , Michael Orgad , Shmuel Ur , Avi Ziv, User defined coverage—a tool supported methodology for design verification, Proceedings of the 35th annual Design Automation Conference, p.158-163, June 15-19, 1998, San Francisco, California, United States [doi>10.1145/277044.277081]
• 7: Yong Woo Kim, Efficient use of code coverage in large-scale software development, Proceedings of the 2003 conference of the Centre for Advanced Studies on Collaborative research, p.145-155, October 06-09, 2003, Toronto, Ontario, Canada
• 8: T. Zimmermann, N. Nagappan, and A. Zeller. Predicting Bugs from History, chapter 4, pages 69--88. Springer, March 2008.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Reliability

Keywords:
coverage analysis, software testing