Evaluating software engineering methods and tool—part 2: selecting an appropriate evaluation method—technical criteria

ABSTRACT

In the previous article about the DESMET evaluation methodology, I identified nine different types of evaluation:1. Quantitative Experiments2. Quantitative Case Studies3. Quantitative Surveys4. Feature Analysis---Screening mode5. Feature Analysis---Case Study6. Feature Analysis---Experiment7. Feature Analysis---Survey8. Qualitative Effects Analysis9. Benchmarking.This article considers the way in which different evaluation requirements and organisation capabilities affect your choice of evaluation method. Your particular choice of evaluation method will be affected by your evaluation goals, the characteristics of the object you want to evaluate, the characteristics of the organisation you work in, and the limitations and constraints placed on the evaluation exercise. These different factors interact in complicated ways, so it is difficult to identify which evaluation method is the most appropriate.The specific criteria that the DESMET methodology uses to determine your circumstances are:1. The evaluation context.2. The nature of the expected impact of using the method/tool.3. The nature of the object (i.e., method/tool/generic method) to be evaluated.4. The scope of impact of the method/tool.5. The maturity of the method/tool.6. The learning curve associated with the method/tool.7. The measurement capability of the organisation undertaking the evaluation.These issues are discussed in the following sections.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
        D.2.10 Design**
            Subjects: Methodologies**

General Terms:
Design, Performance