Talking tests: an empirical assessment of the role of fit acceptance tests in clarifying requirements

ABSTRACT

The starting point for software evolution is usually a change request, expressing the new or updated requirements on the delivered system. The requirements specified in a change request document are often incomplete and inconsistent with the initial requirement document, as well as the implementation. Programmers working on the evolution of the software are often in trouble interpreting an under-specified change request document, resulting in code that does not meet the users' expectations and contains faults that can only be detected later through expensive testing activities.

In this paper, we investigate the role of acceptance tests to clarify the requirements used in software evolution iterations. In particular we focus on Fit tables, a way to express acceptance tests which simplifies their translation into executable test cases. We designed and ran an experiment to assess whether availability of Fit tables affects the level of understanding and the productivity in understanding the requirements. Results indicate that Fit tables significantly improve requirement understanding, but tend to involve additional effort.

REFERENCES

• 1: J. Aarniala. Acceptance testing. In whitepaper. www.cs.helsinki.fi/u/jaarnial/jaarnial-testing.pdf, October 30 2006.
• 2: R. Callan. Building Object-Oriented Systems: An Introduction from Concepts to Implementation in C++. WIT Press (UK); BkDisk edition, 1994.
• 3: N. Juristo and A. Moreno. Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Englewood Cliffs, NJ, 2001.
• 4: G. Melnik, K. Read, and F. Maurer. Suitability of fit user acceptance tests for specifying functional requirements: Developer perspective. In Extreme programming and agile methods -- XP/Agile Universe 2004, pages 60--72, August 2004.
• 5: B. Meyer, On Formalism in Specifications, IEEE Software, v.2 n.1, p.6-26, January 1985 [doi>10.1109/MS.1985.229776]
• 6: Rick Mugridge , Ward Cunningham, Fit for Developing Software: Framework for Integrated Tests (Robert C. Martin), Prentice Hall PTR, Upper Saddle River, NJ, 2005
• 7: A. N. Oppenheim. Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London, 1992.
• 8: K. Read, G. Melnik, and F. Maurer. Examining usage patters of the fit acceptance testing framework. In Proc. 6th International Conference on eXtreme Programming and Agile Processes in Software Engineering (XP2005), pages Lecture Notes in Computer Science, Vol. 3556, Springer Verlag: 127--136 2005, June 18--23 2005.
• 9: Filippo Ricca , Massimiliano Di Penta , Marco Torchiano , Paolo Tonella , Mariano Ceccato, The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes, Proceedings of the 29th international conference on Software Engineering, p.375-384, May 20-26, 2007 [doi>10.1109/ICSE.2007.86]
• 10: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000
• 11: Ralph R. Young, Effective requirements practices, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2001

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.1 Requirements/Specifications

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Experimentation

Keywords:
acceptance testing, empirical studies, fit tables, requirements