Software Reliability Engineering: A Roadmap

ABSTRACT

Software reliability engineering is focused on engineering techniques for developing and maintaining software systems whose reliability can be quantitatively evaluated. In order to estimate as well as to predict the reliability of software systems, failure data need to be properly measured by various means during software development and operational phases. Moreover, credible software reliability models are required to track underlying software failure processes for accurate reliability analysis and forecasting. Although software reliability has remained an active research subject over the past 35 years, challenges and open questions still exist. In particular, vital future goals include the development of new software reliability engineering paradigms that take software architectures, testing techniques, and software failure manifestation mechanisms into consideration. In this paper, we review the history of software reliability engineering, the current trends and existing problems, and specific difficulties. Possible future directions and promising research subjects in software reliability engineering are also addressed.

REFERENCES

• 1: Sousuke Amasaki , Yasunari Takagi , Osamu Mizuno , Tohru Kikuno, A Bayesian Belief Network for Assessing the Likelihood of Fault Content, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.215, November 17-21, 2003
• 2: [2] ANSI/IEEE, Standard Glossary of Software Engineering Terminology, STD-729-1991, ANSI/IEEE, 1991.
• 3: Luciano Baresi , Elisabetta Di Nitto , Carlo Ghezzi, Toward Open-World Software: Issue and Challenges, Computer, v.39 n.10, p.36-43, October 2006 [doi>10.1109/MC.2006.362]
• 4: Antonia Bertolino, Software Testing Research: Achievements, Challenges, Dreams, 2007 Future of Software Engineering, p.85-103, May 23-25, 2007 [doi>10.1109/FOSE.2007.25]
• 5: Judith Bishop , Nigel Horspool, Cross-Platform Development: Software that Lasts, Computer, v.39 n.10, p.26-35, October 2006 [doi>10.1109/MC.2006.337]
• 6: [6] L. Briand and D. Pfahl, "Using Simulation for Assessing the Real Impact of Test Coverage on Defect Coverage," IEEE Transactions on Reliability, vol. 49, no. 1, March 2000, pp. 60-70.
• 7: Jie Cheng , David A. Bell , Weiru Liu, Learning belief networks from data: an information theory based approach, Proceedings of the sixth international conference on Information and knowledge management, p.325-331, November 10-14, 1997, Las Vegas, Nevada, United States [doi>10.1145/266714.266920]
• 8: Xia Cai , Michael R. Lyu, The effect of code coverage on fault detection under different testing profiles, Proceedings of the 1st international workshop on Advances in model-based testing, p.1-7, May 15-21, 2005, St. Louis, Missouri
• 9: [9] X. Cai, M.R. Lyu, and K.F. Wong, "A Generic Environment for COTS Testing and Quality Prediction," Testing Commercial-off-the-shelf Components and Systems, S. Beydeda and V. Gruhn (eds.), Springer-Verlag, Berlin, 2005, pp. 315-347.
• 10: Xia Cai , Michael R. Lyu , Mladen A. Vouk, An Experimental Evaluation on Reliability Features of N-Version Programming, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering, p.161-170, November 08-11, 2005 [doi>10.1109/ISSRE.2005.7]
• 11: Xia Cai , Michael R. Lyu, An Empirical Study on Reliability Modeling for Diverse Software Systems, Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE'04), p.125-136, November 02-05, 2004 [doi>10.1109/ISSRE.2004.6]
• 12: [12] M. Chen, M.R. Lyu, and E. Wong, "Effect of Code Coverage on Software Reliability Measurement," IEEE Transactions on Reliability, vol. 50, no. 2, June 2001, pp.165-170.
• 13: [13] M. H. Chen, A. P. Mathur, and V. J. Rego, "Effect of Testing Techniques on Software Reliability Estimates Obtained Using Time Domain Models," In Proceedings of the 10th Annual Software Reliability Symposium, Denver, Colorado, June 1992, pp. 116-123.
• 14: [14] J. B. Dugan and M. R. Lyu, "Dependability Modeling for Fault-Tolerant Software and Systems," in Software Fault Tolerance, M. R. Lyu (ed.), New York: Wiley, 1995, pp. 109-138.
• 15: [15] D.E. Eckhardt and L.D. Lee, "A Theoretical Basis for the Analysis of Multiversion Software Subject to Coincident Errors," IEEE Transactions on Software Engineering, vol. 11, no. 12, December 1985, pp. 1511-1517.
• 16: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194]
• 17: Joseph R. Horgan , Saul London , Michael R. Lyu, Achieving software quality with testing coverage measures, Computer, v.27 n.9, p.60-69, September 1994 [doi>10.1109/2.312032]
• 18: [18] C. Y. Huang and M. R. Lyu, "Optimal Release Time for Software Systems Considering Cost, Testing-Effort, and Test Efficiency," IEEE Transactions on Reliability, vol. 54, no. 4, December 2005, pp. 583-591.
• 19: Chin-Yu Huang , Michael R. Lyu , Sy-Yen Kuo, A Unified Scheme of Some Nonhomogenous Poisson Process Models for Software Reliability Estimation, IEEE Transactions on Software Engineering, v.29 n.3, p.261-269, March 2003 [doi>10.1109/TSE.2003.1183936]
• 20: [20] W. S. Humphrey, "The Future of Software Engineering: I," Watts New Column, News at SEI, vol. 4, no. 1, March, 2001.
• 21: Valerie Issarny , Mauro Caporuscio , Nikolaos Georgantas, A Perspective on the Future of Middleware-based Software Engineering, 2007 Future of Software Engineering, p.244-258, May 23-25, 2007 [doi>10.1109/FOSE.2007.2]
• 22: Mehdi Jazayeri, Some Trends in Web Application Development, 2007 Future of Software Engineering, p.199-213, May 23-25, 2007 [doi>10.1109/FOSE.2007.26]
• 23: [23] Z. Jelinski and P.B. Moranda, "Software Reliability Research," in Proceedings of the Statistical Methods for the Evaluation of Computer System Performance, Academic Press, 1972, pp. 465-484.
• 24: Bev Littlewood , Lorenzo Strigini, Software reliability and dependability: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.175-188, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336551]
• 25: B. Littlewood , D. R. Miller, Conceptual Modeling of Coincident Failures in Multiversion Software, IEEE Transactions on Software Engineering, v.15 n.12, p.1596-1614, December 1989 [doi>10.1109/32.58771]
• 26: Chao Liu , Long Fei , Xifeng Yan , Jiawei Han , Samuel P. Midkiff, Statistical Debugging: A Hypothesis Testing-Based Approach, IEEE Transactions on Software Engineering, v.32 n.10, p.831-848, October 2006 [doi>10.1109/TSE.2006.105]
• 27: [27] N. Looker and J. Xu, "Assessing the Dependability of SOAP-RPC-Based Web Services by Fault Injection," in Proceedings of 9th IEEE International Workshop on Object-oriented Real-time Dependable Systems, 2003, pp. 163-170.
• 28: Michael R. Lyu, Handbook of software reliability and system reliability, McGraw-Hill, Inc., Hightstown, NJ, 1996
• 29: [29] M. R. Lyu and X. Cai, "Fault-Tolerant Software," Encyclopedia on Computer Science and Engineering, Benjamin Wah (ed.), Wiley, 2007.
• 30: Michael R. Lyu , Zubin Huang , Sam K. S. Sze , Xia Cai, An Empirical Study on Testing and Fault Tolerance for Software Reliability Engineering, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.119, November 17-21, 2003
• 31: [31] Y. K. Malaiya, N. Li, J. M. Bieman, and R. Karcich, "Software Reliability Growth with Test Coverage," IEEE Transactions on Reliability, vol. 51, no. 4, December 2002, pp. 420-426.
• 32: Tiziana Margaria , Bernhard Steffen, Service Engineering: Linking Business and IT, Computer, v.39 n.10, p.45-55, October 2006 [doi>10.1109/MC.2006.355]
• 33: John D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper, Authorhouse, 2004
• 34: John D. Musa, Operational Profiles in Software-Reliability Engineering, IEEE Software, v.10 n.2, p.14-32, March 1993 [doi>10.1109/52.199724]
• 35: John D. Musa , Anthony Iannino , Kazuhira Okumoto, Software reliability: measurement, prediction, application, McGraw-Hill, Inc., New York, NY, 1987
• 36: Hoang Pham, Software Reliability, Springer-Verlag New York, Inc., Secaucus, NJ, 1999
• 37: Peter Popov , Lorenzo Strigini , John May , Silke Kuball, Estimating Bounds on the Reliability of Diverse Systems, IEEE Transactions on Software Engineering, v.29 n.4, p.345-359, April 2003 [doi>10.1109/TSE.2003.1191798]
• 38: Sandra Rapps , Elaine J. Weyuker, Selecting Software Test Data Using Data Flow Information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 39: [39] Rome Laboratory (RL), Methodology for Software Reliability Prediction and Assessment, Technical Report RL-TR-92-52, volumes 1 and 2, 1992.
• 40: Martin L. Shooman, Reliability of Computer Systems and Networks: Fault Tolerance,Analysis,and Design, John Wiley & Sons, Inc., New York, NY, 2002
• 41: Richard N. Taylor , Andre van der Hoek, Software Design and Architecture The once and future focus of software engineering, 2007 Future of Software Engineering, p.226-243, May 23-25, 2007 [doi>10.1109/FOSE.2007.21]
• 42: [42] X. Teng, H. Pham, and D. Jeske, "Reliability Modeling of Hardware and Software Interactions, and Its Applications," IEEE Transactions on Reliability, vol. 55, no. 4, Dec. 2006, pp. 571-577.
• 43: [43] L. A. Tomek and K. S. Trivedi, "Analyses Using Stochastic Reward Nets," in Software Fault Tolerance, M. R. Lyu (ed.), New York: Wiley, 1995, pp. 139-165.
• 44: [44] W. Torres-Pomales, "Software Fault Tolerance: A Tutorial," NASA Langley Research Center, Hampton, Virginia, TM-2000-210616, Oct. 2000.
• 45: Kishor S. Trivedi, SHARPE 2002: Symbolic Hierarchical Automated Reliability and Performance Evaluator, Proceedings of the 2002 International Conference on Dependable Systems and Networks, p.544, June 23-26, 2002
• 46: K. S. Trivedi , K. Vaidyanathan , K. Goseva-Popstojanova, Modeling and Analysis of Software Aging and Rejuvenation, Proceedings of the 33rd Annual Simulation Symposium, p.270, April 16-22, 2000
• 47: [47] A. von Mayrhauser and D. Chen, "Effect of Fault Distribution and Execution Patterns on Fault Exposure in Software: A Simulation Study," Software Testing, Verification & Reliability, vol. 10, no.1, March 2000, pp. 47- 64.
• 48: [48] M. A. Vouk, "Using Reliability Models During Testing With Nonoperational Profiles," in Proceedings of 2nd Bellcore/Purdue Workshop on Issues in Software Reliability Estimation, October 1992, pp. 103-111.
• 49: Wen-Li Wang , Mei-Huei Tang, User-Oriented Reliability Modeling for a Web System, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.293, November 17-21, 2003
• 50: [50] M. Xie, Software Reliability Modeling, World Scientific Publishing Company, 1991.
• 51: [51] S. Yacoub, B. Cukic, and H Ammar, "A Scenario-Based Reliability Analysis Approach for Component-Based Software," IEEE Transactions on Reliability, vol. 53, no. 4, 2004, pp. 465-480.
• 52: Alice X. Zheng , Michael I. Jordan , Ben Liblit , Mayur Naik , Alex Aiken, Statistical debugging: simultaneous identification of multiple bugs, Proceedings of the 23rd international conference on Machine learning, p.1105-1112, June 25-29, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143844.1143983]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability

General Terms:
Reliability, Theory