Comparing software metrics tools

ABSTRACT

This paper shows that existing software metric tools interpret and implement the definitions of object-oriented software metrics differently. This delivers tool-dependent metrics results and has even implications on the results of analyses based on these metrics results. In short, the metrics-based assessment of a software system and measures taken to improve its design differ considerably from tool to tool. To support our case, we conducted an experiment with a number of commercial and free metrics tools. We calculated metrics values using the same set of standard metrics for three software systems of different sizes. Measurements show that, for the same software system and metrics, the metrics values are tool depended. We also defined a (simple) software quality model for "maintainability" based on the metrics selected. It defines a ranking of the classes that are most critical wrt. maintainability. Measurements show that even the ranking of classes in a software system is metrics tool dependent.

REFERENCES

• 1: Jarallah S. Alghamdi , Raimi A. Rufai , Sohel M. Khan, OOMeter: A Software Quality Assurance Tool, Proceedings of the Ninth European Conference on Software Maintenance and Reengineering, p.190-191, March 21-23, 2005 [doi>10.1109/CSMR.2005.44]
• 2: Aqris software. http://www.aqris.com/.
• 3: H. Bär, M. Bauer, O. Ciupke, S. Demeyer, S. Ducasse, M. Lanza, R. Marinescu, R. Nebbe, O. Nierstrasz, M. Przybilski, T. Richner, M. Rieger, C. Riva, A. Sassen, B. Schulz, P. Steyaert, S. Tichelaar, and J. Weisbrod. The FAMOOS Object-Oriented Reengineering Handbook, Oct. 1999.
• 4: Gregory V. Bard, Spelling-error tolerant, order-independent pass-phrases via the damerau-levenshtein string-edit distance metric, Proceedings of the fifth Australasian symposium on ACSW frontiers, January 30-February 02, 2007, Ballarat, Australia
• 5: S. R. Chidamber , C. F. Kemerer, A Metrics Suite for Object Oriented Design, IEEE Transactions on Software Engineering, v.20 n.6, p.476-493, June 1994 [doi>10.1109/32.295895]
• 6: Clarkware consulting inc. http://www.clarkware.com/.
• 7: Fred J. Damerau, A technique for computer detection and correction of spelling errors, Communications of the ACM, v.7 n.3, p.171-176, March 1964 [doi>10.1145/363958.363994]
• 8: R. Geoff Dromey, Cornering the Chimera, IEEE Software, v.13 n.1, p.33-43, January 1996 [doi>10.1109/52.476284]
• 9: EUROCONTROL. Overall Target Architecture Activity (OATA). http://www.eurocontrol.be/oca/public/standard page/overall arch.html, Jan 2007.
• 10: Maurice H. Halstead, Elements of Software Science (Operating and programming systems series), Elsevier Science Inc., New York, NY, 1977
• 11: hello2morrow. http://www.hello2morrow.com/.
• 12: Brian Henderson-Sellers, Object-oriented metrics: measures of complexity, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995
• 13: Watts S. Humphrey, Introduction to the personal software process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 14: hypercision inc., http://hypercision.com/.
• 15: instantiations inc., http://www.instantiations.com/.
• 16: ISO. ISO/IEC 9126-1 "Software engineering - Product Quality - Part 1: Quality model", 2001.
• 17: ISO. ISO/IEC 9126-3 "Software engineering - Product Quality - Part 3: Internal metrics", 2003.
• 18: Andrew cain. http://www.it.swin.edu.au/projects/jmetric/products/jmetric/default.htm.
• 19: Even-André Karlsson, Software reuse: a holistic approach, John Wiley & Sons, Inc., New York, NY, 1995
• 20: N. Kececi and A. Abran. Analysing, Measuring and Assessing Software Quality In a Logic Based Graphical Model, 2001. QUALITA 2001, Annecy, France, 2001, pp. 48--55.
• 21: B. Laguë and A. April. Mapping of Datrix(TM) Software Metrics Set to ISO 9126 Maintainability Sub-Characteristics, October 1996. SES '96, Forum on Software Eng. Standards Issues, Montreal, Canada.
• 22: Young Lee , Kai H. Chang, Reusability and maintainability metrics for object-oriented software, Proceedings of the 38th annual on Southeast regional conference, April 07-08, 2000, Clemson, South Carolina [doi>10.1145/1127716.1127737]
• 23: V. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 1966.
• 24: W. Li and S. Henry. Maintenance Metrics for the Object Oriented Paradigm. In IEEE Proc. of the 1st Int. Sw. Metrics Symposium, pages 52--60, May 1993.
• 25: R. Lincke. Validation of a Standard- and Metric-Based Software Quality Model -- Creating the Prerequisites for Experimentation. Licentiate thesis, MSI, Växjö University, Sweden, Apr 2007.
• 26: R. Lincke and W. Löwe. Compendium of Software Quality Standards and Metrics. http://www.arisa.se/compendium/, 2005.
• 27: J. A. McCall, P. G. Richards, and G. F. Walters. Factors in Software Quality. Technical Report Vol. I, NTIS Springfield, VA, 1977. NTIS AD/A-049 014.
• 28: M squared technologies. http://www.msquaredtechnologies.com/.
• 29: Power software. http://www.powersoftware.com/.
• 30: Semantic designs inc. http://www.semdesigns.com/.
• 31: Walter F. Tichy, Should Computer Scientists Experiment More?, Computer, v.31 n.5, p.32-40, May 1998 [doi>10.1109/2.675631]
• 32: Verifysoft technology. http://www.verifysoft.com/.
• 33: Virtual machinery. http://www.virtualmachinery.com/.
• 34: A. H. Watson and T. J. McCabe. Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric. NIST Special Pub. 500--235, 1996.
• 35: R. K. Yin. Case Study Research: Design and Methods (Applied Social Research Methods). SAGE Publications, December 2002.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics

General Terms:
Measurement, Reliability, Verification

Keywords:
comparing tools, software quality metrics