Using groupings of static analysis alerts to identify files likely to contain field failures

ABSTRACT

In this paper, we propose a technique for leveraging historical field failure records in conjunction with automated static analysis alerts to determine which alerts or sets of alerts are predictive of a field failure. Our technique uses singular value decomposition to generate groupings of static analysis alert types, which we call alert signatures, that have been historically linked to field failure-prone files in previous releases of a software system. The signatures can be applied to sets of alerts from a current build of a software system. Files that have a matching alert signature are identified as having similar static analysis alert characteristics to files with known field failures in a previous release of the system. We performed a case study involving an industrial software system at IBM and found three distinct alert signatures that could be applied to the system. We found that 50% of the field failures reported since the last static analysis run could be discovered by examining the 10% of the files and static analysis alerts indicated by these three alert signatures. The remaining failures were either not detected by a signature which could be an indication of a new type of error in the field, or they were on areas of the code where no static analysis alerts were detected.

REFERENCES

• 1: Brian Chess , Gary McGraw, Static Analysis for Security, IEEE Security and Privacy, v.2 n.6, p.76-79, November 2004 [doi>10.1109/MSP.2004.111]
• 2: IEEE, "IEEE Standard 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology," 1990.
• 3: Nachiappan Nagappan , Thomas Ball, Static analysis tools as early indicators of pre-release defect density, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062558]
• 4: Nachiappan Nagappan , Thomas Ball, Static analysis tools as early indicators of pre-release defect density, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062558]
• 5: Nachiappan Nagappan , Laurie Williams , John Hudepohl , Will Snipes , Mladen Vouk, Preliminary Results On Using Static Analysis Tools For Software Inspection, Proceedings of the 15th International Symposium on Software Reliability Engineering, p.429-439, November 02-05, 2004 [doi>10.1109/ISSRE.2004.30]
• 6: S. Osinski, J. Stefanowski, and D. Weiss, "Lingo: Search Results Clustering Algorithm Based on Singular Value Decomposition," in Advances in Soft Computing, Intelligent Information Processing and Web Mining, Zakopane, Poland, 2004, pp. 359--368.
• 7: T. Will, "Introduction to the Singular Value Decomposition." vol. 2006: UW-La Crosse, 1999.
• 8: J. Zheng, L. Williams, N. Nagappan, W. Snipes, J. Hudepohl, and M. Vouk, "On the Value of Static Analysis for Fault Detection in Software," IEEE Transactions on Software Engineering, vol. 32, no. 4, pp. 240--253, April 2006.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Symbolic execution

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

General Terms:
Experimentation, Management, Measurement, Reliability

Keywords:
field failures, singular value decomposition, static analysis