Producing more reliable software: mature software engineering process vs. state-of-the-art technology?

ABSTRACT

A customer of high assurance software recently sponsored a software engineering experiment in which a real-time software system was developed concurrently by two popular software development methodologies. One company specialized in the state-of-the-practice waterfall method rated at a Capability Maturity Model Level 4. A second developer employed his mathematically based formal method with automatic code generation. As specified in separate contracts, C++ code plus development documentation and process and product metrics (errors) were to be delivered. Both companies were given identical functional specs and agreed to a generous and equal cost, schedule, and explicit functional reliability objectives. At conclusion of the experiment an independent third party determined through extensive statistical testing that neither methodology was able to meet the user's reliability objectives within cost and schedule constraints. The metrics collected revealed the strengths and weaknesses of each methodology and why they were not able to reach customer reliability objectives. This paper will explore the specification for the system under development, the two competing development processes, the products and metrics captured during development, the analysis tools and testing techniques by the third party, and the results of a reliability and process analysis.

REFERENCES

• 1: Larry Apfelbaum, The practical engineer: spec-based tests make sure telecom software works, IEEE Spectrum, v.34 n.11, p.77-83, Nov. 1997 [doi>10.1109/6.632499]
• 2: Haskell, The Haskell 98 Report, http://www.haskell.org/.
• 3: Huang, X., A Comparison Between Standard and Formal Mathematical Development, Master's Thesis, University of Maryland, Department of Nuclear Materials and Reliability Engineering, 1998.
• 4: Kestrel Institute, SPECWARE Users Guide, Version 2.01, Kestrel Institute, 1996.
• 5: Mercury, WinRunner User's Guide, Version 4.0, Mercury Interactive Corporation.
• 6: John D. Musa, Software Reliability Engineering: More Reliable Software Faster and Cheaper, Authorhouse, 2004
• 7: Rose, Rational Rose 98 Edition, http://www.rational.com/ rose/.
• 8: CORPORATE Carnegie Mellon University , Mark C. Paulk , Charles V. Weber , Bill Curtis , Mary Beth Chrissis, The capability maturity model: guidelines for improving the software process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification
          Subjects: Reliability

Additional Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software development; Software process

General Terms:
Design, Experimentation, Management, Measurement, Performance, Reliability, Theory

Keywords:
capability maturity model, formal methods, software engineering experiment, software process and product metrics, software reliability