Software metrics: an introduction and annotated bibliography

ABSTRACT

Software metrics, an area of software engineering, is concerned with various measurements of computer software and its development. Software metrics, its importance, some current areas of investigation, and problems are described. An annotated bibliography of work in software metrics is included.I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in you thoughts, advanced to the stage of science, whatever the matter may be.---Lord Kelvin, Popular Lectures and Addresses, volume 1, 1889

REFERENCES

• 1: {Baker 79} Albert L. Baker and Stuart H. Zweben, "The Use of Software Science in Evaluating Modularity Concepts," IEEE Transactions on Software Engineering, vol. se-5, no. 2, March 1979, pp. 110--120.
• 2: {Baker 80} Albert L. Baker and Stuart H. Zweben, "A Comparison of Measures of Control Flow Complexity," IEEE Transactions on Software Engineering, vol. se-6, no. 6, November 1980, pp. 506--512.
• 3: V. R. Basili , J. C. Knight, A language design for vector machines, ACM SIGPLAN Notices, v.10 n.3, p.39-43, March 1975
• 4: {Basili 79b} Victor R. Basili, "Quantitative Software Complexity Models: A Panel Summary," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 243--245.
• 5: {Basili 79c} Victor R. Basili and Robert W. Reiter, Jr., "Evaluating Automatable Measures of Software Development," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 351--355.
• 6: {Basili 80} Victor R. Basili, Tutorial on Models and Metrics for Software Management and Engineering, Computer Society Press, 1980.
• 7: {Belady 76} L. A. Belady and M. M. Lehman, "A Model of Large Program Development," IBM Systems Journal, vol. 15, no. 3, 1976, pp. 225--252.
• 8: {Belady 78} L. A. Belady, "Measures and Empirical Studies," 2nd Software Life-Cycle Management Workshop, August 1978, pp. 44--46.
• 9: {Belady 79} L. A. Belady, "On Software Complexity," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 90--94.
• 10: {Boehm 73} Barry W. Boehm, "Software and Its Impact: A Quantitative Assessment," Datamation, May 1973, pp. 48--59.
• 11: {Boehm 76} Barry W. Boehm, "Software Engineering," IEEE Transactions on Computers, vol. c-25, no. 12, December 1976, pp. 1226--1241.
• 12: {Boehm 78} Barry W. Boehm, John R. Brown, Hans Kaspar, Myron Lipow, Gordon J. MacLeod, and Michael J. Merritt, Characteristics of Software Quality, North Holland, 1978.
• 13: Barry W. Boehm, Software engineering-as it is, Proceedings of the 4th international conference on Software engineering, p.11-21, September 17-19, 1979, Munich, Germany
• 14: {Boulton 79} P. I. P. Boulton and M. A. R. Kittler, "Estimating Program Reliability," The Computer Journal, vol. 22, no. 4, November 1979, pp. 328--331.
• 15: Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Softw, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1978
• 16: Ruven E. Brooks, Studying programmer behavior experimentally: the problems of proper methodology, Communications of the ACM, v.23 n.4, p.207-213, April 1980 [doi>10.1145/358841.358847]
• 17: John R. Brown , Kurt F. Fischer, A graph theoretic approach to the verification of program structures, Proceedings of the 3rd international conference on Software engineering, p.136-141, May 10-12, 1978, Atlanta, Georgia, United States
• 18: R. N. Chanon, On a measure of program structure, Programming Symposium, Proceedings Colloque sur la Programmation, p.9-16, April 09-11, 1974
• 19: {Chen 78} Edward T. Chen, "Program Complexity and Programmer Productivity," IEEE Transactions on Software Engineering, vol. se-4, no. 3, May 1978, pp. 187--194.
• 20: {Cheung 80} Roger C. Cheung, "A User-Oriented Software Reliability Model," IEEE Transactions on Software Engineering, vol. se-6, no. 2, March 1980, pp. 118--125.
• 21: {Curtis 79a} Bill Curtis, Sylvia B. Sheppard, Phil Milliman, M. A. Borst, and Tom Love, "Measuring the Psychological Complexity of Software Maintenance Tasks with the Halstead and McCabe Metrics," IEEE Transactions on Software Engineering, vol. se-5, no. 2, March 1979, pp. 96--104.
• 22: Bill Curtis , Sylvia B. Sheppard , Phil Milliman, Third time charm: Stronger prediction of programmer performance by software complexity metrics, Proceedings of the 4th international conference on Software engineering, p.356-360, September 17-19, 1979, Munich, Germany
• 23: {Curtis 79c} Bill Curtis, "In Search of Software Complexity," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 95--106.
• 24: {Curtis 80} Bill Curtis, "Measurement and Experimentation in Software Engineering," Proceedings of the IEEE, vol. 68, no. 9, September 1980, pp. 1144--1157.
• 25: {De Marco79} Tom De Marco, Concise Notes on Software Engineering, Yourdon, 1979.
• 26: Edsger W. Dijkstra, The humble programmer, Communications of the ACM, v.15 n.10, p.859-866, Oct. 1972 [doi>10.1145/355604.361591]
• 27: Edsger W. Dijkstra, My hopes of computing science (EWD709), Proceedings of the 4th international conference on Software engineering, p.442-448, September 17-19, 1979, Munich, Germany
• 28: {Di Persio 80} Tom Di Persio, Dan Isbister, and Ben Shneiderman, "An Experiment Using Memorization/Reconstruction as a Measure of Programmer Ability," International Journal of Man-Machine Studies, vol. 13, 1980, pp. 339--354.
• 29: {Dunsmore 80} H. E. Dunsmore and J. D. Gannon, "Analysis of the Effects of Programming Factors on Programming Effort," The Journal of systems and Software, vol. 1, no. 2, 1980, pp. 141--153.
• 30: James L. Elshoff, Measuring commercial PL/I programs using Halstead's criteria, ACM SIGPLAN Notices, v.11 n.5, p.38-46, May 1976 [doi>10.1145/956003.956006]
• 31: {Elshoff 76b} James L. Elshoff, "An Analysis of Some Commercial PL/I Programs," IEEE Transactions on Software Engineering, vol. se-2, no. 2, June 1976, pp. 113--120.
• 32: James L. Elshoff, An investigation into the effects of the counting method used on software science measurements, ACM SIGPLAN Notices, v.13 n.2, p.30-45, February 1978 [doi>10.1145/953422.953423]
• 33: James L. Elshoff, A study of the structural composition of PL/I programs, ACM SIGPLAN Notices, v.13 n.6, June 1978 [doi>10.1145/987515.987518]
• 34: {Endres 75} Albert Endres, "An Analysis of Errors and Their Causes in System Programs," IEEE Transactions on Software Engineering, vol. se-1, no. 2, June 1975, pp. 140--149.
• 35: Alan R. Feuer , Edward B. Fowlkes, Some results from an empirical study of computer software, Proceedings of the 4th international conference on Software engineering, p.351-355, September 17-19, 1979, Munich, Germany
• 36: {Fitsos 80} George P. Fitsos, "Vocabulary Effects in Software Science," Compsac 80, October 1980, pp. 751--756.
• 37: {Fitzsimmons 78a} Ann Bowman Fitzsimmons, "Relating the Presence of Software Errors to the Theory of Software Science," 11th Hawaii International Conference on System Sciences, vol. 1, 1978, pp. 40--46.
• 38: Ann Fitzsimmons , Tom Love, A Review and Evaluation of Software Science, ACM Computing Surveys (CSUR), v.10 n.1, p.3-18, March 1978 [doi>10.1145/356715.356717]
• 39: {Gilb 75} Tom Gilb, "Software Metrics: State of the Art," Computer Weekly, September 11, 1975, p. 6.
• 40: {Gilb 77} Tom Gilb, Software Metrics, Winthrop, 1977.
• 41: {Gordon 79a} Ronald D. Gordon, "Measuring Improvements in Program Clarity," IEEE Transactions on Software Engineering, vol. se-5, no. 2, March 1979, pp. 79--90.
• 42: {Gordon 79b} Ronald D. Gordon, "A Qualitative Justification for a Measure of Program Clarity," IEEE Transactions on Software Engineering, vol. se-5, no. 2, March 1979, pp. 121--128.
• 43: Maurice H. Halstead, Elements of Software Science (Operating and programming systems series), Elsevier Science Inc., New York, NY, 1977
• 44: {Halstead 79} M. H. Halstead, "Advances in Software Science," Advances in Computers, vol. 18, Academic Press, 1979, pp. 119--172.
• 45: {Halstead 80} M. H. Halstead and Victor Schneider, "Self-Assessment Procedure VII," Communications of the ACM, vol. 23, no. 8, August 1980, pp. 475--480.
• 46: Patricia A. Hamilton , John D. Musa, Measuring reliability of computation center software, Proceedings of the 3rd international conference on Software engineering, p.29-36, May 10-12, 1978, Atlanta, Georgia, United States
• 47: Wilfred J. Hansen, Measurement of program complexity by the pair: (Cyclomatic Number, Operator Count), ACM SIGPLAN Notices, v.13 n.3, p.29-33, March 1978 [doi>10.1145/954373.954375]
• 48: C. A. R. Hoare, Software engineering: A keynote address, Proceedings of the 3rd international conference on Software engineering, p.1-4, May 10-12, 1978, Atlanta, Georgia, United States
• 49: {Jelinski 72} Z. Jelinski and P. Moranda, "Software Reliability Research," Statistical Computer Performance Evaluation, Walter Freiberger, ed., Academic Press, 1972, pp. 465--484.
• 50: Dan B. Johnston , Andrew M. Lister, An Experiment in Software Science, Proceedings of a Symposium on Language Design and Programming Methodology, p.195-215, September 10-11, 1979
• 51: {Jones 78} T. C. Jones, "Measuring Programming Quality and Productivity," IBM Systems Journal, vol. 17, no. 1, 1978, pp. 39--63.
• 52: {Jørgensen 80} Anker Helms Jørgensen, "A Methodology for Measuring the Readability and Modifiability of Computer Programs," BIT, vol. 20, 1980, pp. 394--405.
• 53: {Knijff 78} D. J. J. van tier Knijff, "Software Physics and Program Analysis," The Australian Computer Journal, vol. 10, no. 3, August 1978, pp. 82--86.
• 54: {Knijff 79} D. van der Knijff and J.-L. Lassez, "A Clarification of the Comparison Between Some Measures of Software Science," Information Processing Letters, vol. 8, no. 5, June 11, 1979, pp. 241--243.
• 55: {Kolence 75} Kenneth W. Kolence, "Software Physics," Datamation, June 1975, pp. 48--51.
• 56: Kenneth W. Kolence, Introduction to Software Physics, McGraw-Hill, Inc., New York, NY, 1985
• 57: {Kolence 78} Kenneth W. Kolence, "Software Physics Workload Characterization," Performance of Computer Installations, D. Ferrari, ed., North Holland, 1978, pp. 333--341.
• 58: {Lassez 79} J.-L. Lassez and D. van der Knijff, "Evaluation of Length and Level for Simple Program Schemes," Compsac 79, November 1979, pp. 688--694.
• 59: {Levy 79} Dave Levy, Don Guy, and Jim Ronback, "A Place for 'Metrics' in Software Development," Telesis, October 1979, pp. 17--22.
• 60: {Littlewood 73} Bev Littlewood and J. L. Verrall, "A Bayesian Reliability Growth Model for Computer Software," Journal of the Royal Statistical Sodety (series C), Applied Statistics, vol. 22, no. 3, 1973, pp. 332--346.
• 61: B. Littlewood, How to measure software reliability, and how not to, Proceedings of the 3rd international conference on Software engineering, p.37-45, May 10-12, 1978, Atlanta, Georgia, United States
• 62: {Littlewood 78b} Bev Littlewood, "Validation of a Software Reliability Model," 2nd Software Life Cycle Management Workshop, August 1978, pp. 146--152.
• 63: L. T. Love , A. B. Bowman, An independent test of the theory of software physics, ACM SIGPLAN Notices, v.11 n.11, November 1976 [doi>10.1145/987335.987340]
• 64: Tom Love, An experimental investigation of the effect of program structure on program understanding, ACM SIGPLAN Notices, v.12 n.3, p.105-113, March 1977
• 65: {McCabe 76} Thomas J. McCabe, "A Complexity Measure," IEEE Transactions on Software Engineering, vol. se-2, no. 4, December 1976, pp. 308--320.
• 66: {McCall 80} James A. McCall, "An Assessment of Current Software Metric Research," EASCON'80, 1980, pp. 323--333.
• 67: Carma L. McClure, A model for program complexity analysis, Proceedings of the 3rd international conference on Software engineering, p.149-157, May 10-12, 1978, Atlanta, Georgia, United States
• 68: {Mills 76} Harlan D. Mills, "Software Development," IEEE Transactions on Software Engineering, vol. se-2, no. 4, December 1976, pp. 265--273.
• 69: Isao Miyamoto, Toward an effective software reliability evaluation, Proceedings of the 3rd international conference on Software engineering, p.46-55, May 10-12, 1978, Atlanta, Georgia, United States
• 70: Siba N. Mohanty, Models and Measurements for Quality Assessment of Software, ACM Computing Surveys (CSUR), v.11 n.3, p.251-275, Sept. 1979 [doi>10.1145/356778.356783]
• 71: {Moranda 78} Paul B. Moranda, "The (Sad) Status of; (Unapproached) Limits to; and (Manifold) Alternatives for Software Measurement Techniques," IEEE Spring Compcon 78, February 1978, pp. 353--354.
• 72: {Musa75} John D. Musa, "A Theory of Software Reliability and Its Application," IEEE Transactions on Software Engineering, vol. se-1, no. 3, September 1975, pp. 312--327.
• 73: {Musa 80a} John D. Musa, "Software Reliability Measurement," The Journal of Systems and Software, vol. 1, no. 3, 1980, pp. 223--241.
• 74: {Musa 80b} John D. Musa, "The Measurement and Management of Software Reliability," Proceedings of the IEEE, vol. 68, no. 9, September 1980, pp. 1131--1143.
• 75: Glenford J. Myers, An extension to the cyclomatic measure of program complexity, ACM SIGPLAN Notices, v.12 n.10, p.61-64, October 1977 [doi>10.1145/954627.954633]
• 76: {Norden 77} Peter V. Norden, "Project Life Cycle Modelling: Background and Application of the Life Cycle Curves," First Software Life Cycle Management Workshop, August 1977, pp. 217--306.
• 77: {Oldehoeft 77} R. R. Oldehoeft, "A Contrast Between Language Level Measures," IEEE Transactions on Software Engineering, vol. se-3, no. 6, November 1977, pp. 476--478.
• 78: {Oldehoeft 79} R. R. Oldehoeft and Leonard J. Bass, "Dynamic Software Science with Applications," IEEE Transactions on Software Engineering, vol. se-5, no. 5, September 1979, pp. 497--504.
• 79: K. J. Ottenstein, An algorithmic approach to the detection and prevention of plagiarism, ACM SIGCSE Bulletin, v.8 n.4, p.30-41, Dec. 1976 [doi>10.1145/382222.382462]
• 80: {Ottenstein, L. 79} Linda M. Ottenstein, "Quantitative Estimates of Debugging Requirements," IEEE Transactions on Software Engineering, vol. se-5, no. 5, September 1979, pp. 504--514.
• 81: {Parr 80} F. N. Parr, "An Alternative to the Rayleigh Curve Model for Software Development Effort," IEEE Transactions on Software Engineering, vol. se-6, no. 3, May 1980, pp. 291--296.
• 82: {Putnam 80a} Lawrence Putnam, Tutorial on Software Cost Estimating and Life-Cycle Control: Getting the Software Numbers, Computer Society Press, 1980.
• 83: {Putnam 80b} Lawrence H. Putnam, "The Real Metrics of Software Development," EASCON'80, 1980, pp. 310--322.
• 84: {Randell 76} B. Randell, P. Naur, J. N. Buxton, Software Engineering: Concepts and Techniques, Petroceμi-Charter, 1976.
• 85: B. Randell, Software engineering in 1968, Proceedings of the 4th international conference on Software engineering, p.1-10, September 17-19, 1979, Munich, Germany
• 86: Raymond J. Rubey , R. Dean Hartwick, Quantitative measurement of program quality, Proceedings of the 1968 23rd ACM national conference, p.671-677, August 27-29, 1968 [doi>10.1145/800186.810631]
• 87: Victor Schneider, Prediction of software effort and project duration: four new formulas, ACM SIGPLAN Notices, v.13 n.6, June 1978 [doi>10.1145/987515.987520]
• 88: {Schneidewind 79} N. F. Schneidewind, "Software Metrics for Aiding Program Development and Debugging," AFIPS Conference Proceedings, vol. 48, 1979, pp. 989--994.
• 89: {Shen 79} V. Y. Shen, "The Relationship Between Student Grades and Software Science Parameters," Compsac 79, November 1979, pp. 783--787.
• 90: {Sheperd 80} J. A. Sheperd and J.-L. Lassez, "Opposing Views on the Use of Software Science Measures for the Automatic Assessment of Student Programs," Australian Computer Science Communications, vol. 2, no. 1, January 1980, pp. 205--215.
• 91: {Sheppard 80} Sylvia B. Sheppard, Phil Milliman, and Bill Curtis, "Experimental Evaluation of On-Line Program Construction," Compsac 80, October 1980, pp. 505--510.
• 92: {Shneiderman 77} B. Shneiderman, "Measuring Computer Program Quality and Comprehension," International Journal of Man-Machine Studies, vol. 9, 1977, pp. 465--478.
• 93: {Shooman 79} Martin L. Shooman, "Tutorial on Software Cost Models," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 1--19.
• 94: {Sime 73} M. E. Sime, T. R. G. Green, and D. J. Guest, "Psychological Evaluation of Two Conditional Construction Used in Computer Languages," International Journal of Man-Machine studies, vol. 5, 1973, pp. 105--113.
• 95: {Stevens76} Barry A. Stevens, "'Software Physics' Standardizes DP Operations," Computerworld, September 20, 1976, pp. 15, 18.
• 96: Murat M. Tanik, A comparison of program complexity prediction models, ACM SIGSOFT Software Engineering Notes, v.5 n.4, p.10-16, October 1980 [doi>10.1145/1010884.1010888]
• 97: {Thayer 78} T. A. Thayer, M. Lipow, and E. C. Nelson, Software Reliability: A Study of Large Project Reality, North Holland, 1978.
• 98: Wladyslaw M. Turski, Look ahead at software engineering, Proceedings of the 4th international conference on Software engineering, p.449-456, September 17-19, 1979, Munich, Germany
• 99: {Vemuri 79} V. Vemuri, "The Current State of Software Reliability Modeling," Workshop on Quantitative Software Models for Reliability, Complexity, and Cost, October 1979, pp. 232--238.
• 100: {Walston 77} C. E. Walston and C. P. Felix, "A Method of Programming Measurement and Estimation," IBM Systems Journal, vol. 16, no. 1, 1977, pp. 54--73.
• 101: G. M. Weinberg, The psychology of computer programming, Van Nostrand Reinhold Co., New York, NY, 1988
• 102: {Weiss 79} David M. Weiss, "Evaluating Software Development by Error Analysis: The Data from the Architecture Research Facility," The Journal of Systems and Software, vol. 1, no. 1, 1979, pp. 57--70.
• 103: Larry Weissman, Psychological complexity of computer programs: an experimental methodology, ACM SIGPLAN Notices, v.9 n.6, p.25-36, June 1974 [doi>10.1145/953233.953237]
• 104: {Woodfield 79} Scott N. Woodfield, "An Experiment on Unit Increase in Problem Complexity," IEEE Transactions on Software Engineering, vol. se-5, no. 2, March 1979, pp. 76--79.
• 105: {Woodward 79} Martin R. Woodward, Michael A. Hennell, and David Hedley, "A Measure of Control Flow Complexity in Program Text," IEEE Transactions on Software Engineering, vol. se-5, no. 1, January 1979, pp. 45--50.
• 106: Marvin V. Zelkowitz, Perspectives in Software Engineering, ACM Computing Surveys (CSUR), v.10 n.2, p.197-216, June 1978 [doi>10.1145/356725.356731]
• 107: {Zolnowski 77} Jean M. Zolnowski and Dick B. Simmons, 'Measuring Program Complexity," IEEE Fall Compcon 77, September 1977, pp. 336--340.
• 108: {Zweben 79} Stuart H. Zweben and Kin-Chee Fung, "Exploring Software Science Relations in COBOL and APL," Compsac 79, November 1979, pp. 702--707.