Observations and lessons learned from automated testing

ABSTRACT

This report addresses some of our observations made in a dozen of projects in the area of software testing, and more specifically, in automated testing. It documents, analyzes and consolidates what we consider to be of interest to the community. The major findings can be summarized in a number of lessons learned, covering test strategy, testability, daily integration, and best practices.The report starts with a brief description of five sample projects. Then, we discuss our observations and experiences and illustrate them with the sample projects. The report concludes with a synopsis of these experiences and with suggestions for future test automation endeavors.

REFERENCES

• 1: Stefan Berner, About the development of a point of sale system: an experience report, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 2: Frederick P. Brooks, Jr., No silver bullet: essence and accidents of software engineering, Computer, v.20 n.4, p.10-19, April 1987 [doi>10.1109/MC.1987.1663532]
• 3: Mohammed Elkoutbi , Ismaïl Khriss , Rudolf K. Keller, Automated Prototyping of User Interfaces Based on UML Scenarios, Automated Software Engineering, v.13 n.1, p.5-40, January 2006 [doi>10.1007/s10515-006-5465-5]
• 4: Mark Fewster , Dorothy Graham, Software test automation: effective use of test execution tools, ACM Press/Addison-Wesley Publishing Co., New York, NY, 1999
• 5: http://www.cenqua.com/clover/
• 6: http://www.clarkware.com/software/JUnitPerf.html
• 7: http://www.quest.com/jprobe/
• 8: Kaner, C. Architectures of Test Automation. Software Testing, Analysis & Review Conference (Star) West, San Jose, CA, Oct 2000.
• 9: Kaner, C. Improving the Maintainability of Automated Test Suites. Software QA, 4(4), 1997.
• 10: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 11: Pettichord, B. Design for Testability. Proc. of Pacific Northwest Software Quality Conference, Oct 2002.
• 12: Pettichord, B. Seven Steps to Test Automation Success. http://www.pettichord.com (Revised version of a paper presented at STAR West, San Jose, Nov 1999).
• 13: Gerald M. Weinberg, Quality software management (Vol. 1): systems thinking, Dorset House Publishing Co., Inc., New York, NY, 1992

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.3 Coding Tools and Techniques

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Keywords:
automated testing, software test, test management