Predictors of customer perceived software quality

ABSTRACT

Predicting software quality as perceived by a customer may allow an organization to adjust deployment to meet the quality expectations of its customers, to allocate the appropriate amount of maintenance resources, and to direct quality improvement efforts to maximize the return on investment. However, customer perceived quality may be affected not simply by the software content and the development process, but also by a number of other factors including deployment issues, amount of usage, software platform, and hardware configurations. We predict customer perceived quality as measured by various service interactions, including software defect reports, requests for assistance, and field technician dispatches using the afore mentioned and other factors for a large telecommunications software system. We employ the non-intrusive data gathering technique of using existing data captured in automated project monitoring and tracking systems as well as customer support and tracking systems. We find that the effects of deployment schedule, hardware configurations, and software platform can increase the probability of observing a software failure by more than 20 times. Furthermore, we find that the factors affect all quality measures in a similar fashion. Our approach can be applied at other organizations, and we suggest methods to independently validate and replicate our results.

REFERENCES

• 1: David L. Atkins , Thomas Ball , Todd L. Graves , Audris Mockus, Using version control data to evaluate the impact of software tools: a case study of the version editor, IEEE Transactions on Software Engineering, v.28 n.7, p.625-637, July 2002 [doi>10.1109/TSE.2002.1019478]
• 2: N. Breslow. Statistics in epidemiology: the case control study. JASA, 91(433):14--28, 1996.
• 3: M. Buckley , R. Chillarege, Discovering relationships between service and customer satisfaction, Proceedings of the International Conference on Software Maintenance, p.192, October 17-20, 1995
• 4: R. Chillarege , S. Biyani , J. Rosenthal, Measurement of Failure Rate in Widely Distributed Software, Proceedings of the Twenty-Fifth International Symposium on Fault-Tolerant Computing, p.424, June 27-30, 1995
• 5: S. Chulani. Coqualmo (constructive quality model) a software defect density prediction model. Project Control for Software Quality, 1999.
• 6: S. Chulani, P.Santhanam, D. Moore, and G. Davidson. Deriving a software quality view from customer satisfaction and service data. European Conference on Metrics and Measurement, 2001.
• 7: S. R. Dalal and C. L. Mallows. When should one stop testing software? Journal of American Statist. Assoc, 83:872--879, 1988.
• 8: Todd L. Graves , Alan F. Karr , J. S. Marron , Harvey Siy, Predicting Fault Incidence Using Software Change History, IEEE Transactions on Software Engineering, v.26 n.7, p.653-661, July 2000 [doi>10.1109/32.859533]
• 9: Wendell D. Jones , John P. Hudepohl , Taghi M. Khoshgoftaar , Edward B. Allen, Application of a Usage Profile in Software Quality Models, Proceedings of the Third European Conference on Software Maintenance and Reengineering, p.148, March 03-05, 1999
• 10: P. L. Li, M. Shaw, K. Stolarick, and K. Wallnau. The potential for synergy between certification and insurance. International Workshop on Reuse Economics in conjunction with ICSR7, April 2002.
• 11: M. R. Lyu. Handbook of Software Reliability Engineering. IEEE Society Press, Los Alamitos, CA, 1996.
• 12: A. Mockus and D. M. Weiss. Predicting risk of software changes. Bell Labs Technical Journal, 5(2):169--180, April-June 2000.
• 13: Audris Mockus , David M. Weiss , Ping Zhang, Understanding and predicting effort in software projects, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 14: John D. Musa , Anthony Iannino , Kazuhira Okumoto, Software reliability: measurement, prediction, application, McGraw-Hill, Inc., New York, NY, 1987
• 15: Jeffrey M. Voas, User Participation-based Software Certification, Collected papers from the 5th European Symposium on Validation and Verification of Knowledge Based Systems - Theory, Tools and Practice, p.267-276, June 09-11, 1999
• 16: S. Weisberg. Applied Linear Regression, 2nd Edition. John Wiley & Sons, USA, 1985.
• 17: Alan P. Wood, Software Reliability from the Customer View, Computer, v.36 n.8, p.37-42, August 2003 [doi>10.1109/MC.2003.1220580]
• 18: P. Zhang, J. Landwehr, and M. Serban. Quantifying the value of remote maintenance: An analysis of customer outage data, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures; Complexity measures

Keywords:
metrics, modeling, quality