Search based data sensitivity analysis applied to requirement engineering

ABSTRACT

Software engineering is plagued by problems associated with unreliable cost estimates. This paper introduces an approach to sensitivity analysis for requirements engineering. It uses Search-Based Software Engineering to aid the decision maker to explore sensitivity of the cost estimates of requirements for the Next Release Problem (NRP). The paper presents both single- and multi-objective formulation of NRP with empirical sensitivity analysis on synthetic and real-world data. The results show strong correlation between the level of inaccuracy and the impact on the selection of requirements, as well as between the cost of requirements and the impact, which is as intuitively expected. However, there also exist a few sensitive exceptions to these trends; the paper uses a heat-map style visualisation to reveal these exceptions which require careful consideration. The paper also shows that such unusually sensitivity patterns occur in real-world data and how the proposed approach clearly identifies them.

REFERENCES

• 1: Phillip Armour, Ten unmyths of project estimation, Communications of the ACM, v.45 n.11, November 2002 [doi>10.1145/581571.581582]
• 2: Bagnall, A., Rayward-Smith, V., and Whittley, I. The Next Release Problem. Information and Software Technology 43, 14 (2001), 883--890.
• 3: Paul Baker , Mark Harman , Kathleen Steinhofel , Alexandros Skaliotis, Search Based Approaches to Component Selection and Prioritization for the Next Release Problem, Proceedings of the 22nd IEEE International Conference on Software Maintenance, p.176-185, September 24-27, 2006 [doi>10.1109/ICSM.2006.56]
• 4: Banjotopoulos, C. A Contribution to the Sensitivity Analysis of the Sea-Bed-Structure Interaction Problem for Underwater Pipelines. Computers&structures 40, 6 (1991), 1421--1427.
• 5: Barry Boehm , Chris Abts , Sunita Chulani, Software development cost estimation approaches – A survey, Annals of Software Engineering, v.10 n.1-4, p.177-205, 2000 [doi>10.1023/A:1018991717352]
• 6: Barry W. Boehm, Software Engineering Economics, Prentice Hall PTR, Upper Saddle River, NJ, 1981
• 7: Christopher Frey, H., and Patil, S. Identification and Review of Sensitivity Analysis Methods. Risk Analysis 22, 3 (2002), 553--578.
• 8: Deb, K., and Gupta, H. Searching for Robust Pareto-Optimal Solutions in Multi-Objective Optimization. In Conference on Evolutionary Multi-Criterion Optimization (EMO'05) (2005), vol. 3410, Springer, pp. 150--164.
• 9: Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. Evolutionary Computation, IEEE Transactions on 6, 2 (2002), 182--197.
• 10: Flyvbjerg, B., Holm, M.K.S., and Buhl, S.L. Underestimating Costs in Public Works Projects. Journal of the American Planning Association 68, 3 (2002), 279--295.
• 11: Flyvbjerg, B., Holm, M.K.S., and Buhl, S.L. How (In)ccurate Are Demand Forecasts in Public Works Projects? Journal of the American Planning Association 71, 2 (2005), 131--146.
• 12: Gunawan, R., Cao, Y., Petzold, L., and Doyle, F. Sensitivity Analysis of Discrete Stochastic Systems. Biophysical Journal 88, 4 (2005), 2530--2540.
• 13: Levine, R., and Renelt, D. A Sensitivity Analysis of Cross-Country Growth Regressions. American Economic Review 82, 4 (September 1992), 942--63.
• 14: Osyczka, A. Multicriteria Optimization for Engineering Design. In Design Optimization (1985), pp. 193--227.
• 15: Saltelli, A., Chan, K., Scott, E., et al. Sensitivity Analysis. Wiley Chichester, 2000.
• 16: Marjan van den Akker , Sjaak Brinkkemper , Guido Diepen , Johan Versendaal, Software product release planning through optimization and what-if analysis, Information and Software Technology, v.50 n.1-2, p.101-111, January, 2008 [doi>10.1016/j.infsof.2007.10.017]
• 17: David A. van Veldhuizen , Gary B. Lamont, Multiobjective evolutionary algorithm test suites, Proceedings of the 1999 ACM symposium on Applied computing, p.351-357, February 28-March 02, 1999, San Antonio, Texas, United States [doi>10.1145/298151.298382]
• 18: Yuanyuan Zhang , Mark Harman , S. Afshin Mansouri, The multi-objective next release problem, Proceedings of the 9th annual conference on Genetic and evolutionary computation, July 07-11, 2007, London, England [doi>10.1145/1276958.1277179]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications
          Subjects: Methodologies (e.g., object-oriented, structured)

General Terms:
Algorithms, Experimentation, Management

Keywords:
data sensitivity analysis, empirical study, multi-objective genetic algorithms, multi-objective optimisation, pareto optimality, the next release problem