Interval quality: relating customer-perceived quality to process quality

ABSTRACT

We investigate relationships among software quality measures commonly used to assess the value of a technology, and several aspects of customer perceived quality measured by Interval Quality (IQ): a novel measure of the probability that a customer will observe a failure within a certain interval after software release. We integrate information from development and customer support systems to compare defect density measures and IQ for six releases of a major telecommunications system. We find a surprising negative relationship between the traditional defect density and IQ. The four years of use in several large telecommunication products demonstrates how a software organization can control customer perceived quality not just during development and verification, but also during deployment by changing the release rate strategy and by increasing the resources to correct field problems rapidly. Such adaptive behavior can compensate for the variations in defect density between major and minor releases.

REFERENCES

• 1: Victor R. Basili , Lionel C. Briand , Walcélio L. Melo, A Validation of Object-Oriented Design Metrics as Quality Indicators, IEEE Transactions on Software Engineering, v.22 n.10, p.751-761, October 1996 [doi>10.1109/32.544352]
• 2: D. M. Weiss , V. R. Basili, Evaluating Software Development by Analysis of Changes: Some Data from the Software Engineering Laboratory, IEEE Transactions on Software Engineering, v.11 n.2, p.157-168, February 1985 [doi>10.1109/TSE.1985.232190]
• 3: M. Buckley , R. Chillarege, Discovering relationships between service and customer satisfaction, Proceedings of the International Conference on Software Maintenance, p.192, October 17-20, 1995
• 4: D. A. Christenson and S. T. Huang. Estimating the fault content of software using the fix-on-fix model. Bell Labs Technical Journal, 1(1):130--137, Summer 1996.
• 5: S. Chulani. Coqualmo (constructive quality model) a software defect density prediction model. Project Control for Software Quality, 1999.
• 6: S. Chulani, P.Santhanam, D. Moore, and G. Davidson. Deriving a software quality view from customer satisfaction and service data. European Conference on Metrics and Measurement, 2001.
• 7: Stephen G. Eick , Clive R. Loader , M. David Long , Lawrence G. Votta , Scott Vander Wiel, Estimating software fault content before coding, Proceedings of the 14th international conference on Software engineering, p.59-65, May 11-15, 1992, Melbourne, Australia [doi>10.1145/143062.143090]
• 8: Norman Fenton , Shari Lawrence Pfleeger, Software metrics (2nd ed.): a rigorous and practical approach, PWS Publishing Co., Boston, MA, 1997
• 9: Robert B. Grady , Deborah L. Caswell, Software metrics: establishing a company-wide program, Prentice-Hall, Inc., Upper Saddle River, NJ, 1987
• 10: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 11: M. H. Halstead. Elements of Software Science. Elsevier - North Holland, 1979.
• 12: J. Jelinski and P. B. Moranda. Software reliability research. In W. Freiberger, editor, Probabilistic Models for Software, pages 485--502. Academic Press, 1972.
• 13: T. J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering, v.2 n.4, p.308-320, July 1976 [doi>10.1109/TSE.1976.233837]
• 14: A. K. Midha. Software configuration management for the 21st century. Bell Labs Technical Journal, 2(1), Winter 1997.
• 15: A. Mockus and D. M. Weiss. Predicting risk of software changes. Bell Labs Technical Journal, 5(2):169--180, April-June 2000.
• 16: Audris Mockus , David M. Weiss , Ping Zhang, Understanding and predicting effort in software projects, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 17: Audris Mockus , Ping Zhang , Paul Luo Li, Predictors of customer perceived software quality, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062506]
• 18: Siba N. Mohanty, Models and Measurements for Quality Assessment of Software, ACM Computing Surveys (CSUR), v.11 n.3, p.251-275, Sept. 1979 [doi>10.1145/356778.356783]
• 19: J. Munson , Y. M. Khoshgoftaar, Regression modelling of software quality: empirical investigation, Journal of Electronic Materials, v.19 n.6, p.106-114, Jun. 1990
• 20: John D. Musa , Anthony Iannino , Kazuhira Okumoto, Software reliability: measurement, prediction, application, McGraw-Hill, Inc., New York, NY, 1987
• 21: Niclas Ohlsson , Hans Alberg, Predicting Fault-Prone Software Modules in Telephone Switches, IEEE Transactions on Software Engineering, v.22 n.12, p.886-894, December 1996 [doi>10.1109/32.553637]
• 22: M. Rochkind. The source code control system. IEEE Trans. on Software Engineering, 1(4):364--370, 1975.
• 23: G. J. Schick , R. W. Wolverton, An Analysis of Competing Software Reliability Models, IEEE Transactions on Software Engineering, v.4 n.2, p.104-120, March 1978 [doi>10.1109/TSE.1978.231481]
• 24: N. F. Schneidewind , H. -M. Hoffmann, An Experiment in Software Error Data Collection and Analysis, IEEE Transactions on Software Engineering, v.5 n.3, p.276-286, May 1979 [doi>10.1109/TSE.1979.234188]
• 25: Vincent Y. Shen , Tze-Jie Yu , Stephen M. Thebaut , Lorri R. Paulsen, Identifying Error-Prone Software An Empirical Study, IEEE Transactions on Software Engineering, v.11 n.4, p.317-324, April 1985 [doi>10.1109/TSE.1985.232222]
• 26: D. Weiss. Evaluating software development by error analysis: The data from the architecture research facility. J. Systems and Software, 1:57--70, 1979.
• 27: David M. Weiss , David Bennett , John Y. Payseur , Pat Tendick , Ping Zhang, Goal-oriented software assessment, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581369]
• 28: T.-J. Yu , V. Y. Shen , H. E. Dunsmore, An Analysis of Several Software Defect Models, IEEE Transactions on Software Engineering, v.14 n.9, p.1261-1270, September 1988 [doi>10.1109/32.6170]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Process metrics; Product metrics

General Terms:
Measurement, Reliability

Keywords:
software metrics