Software Testing Research: Achievements, Challenges, Dreams

ABSTRACT

Software engineering comprehends several disciplines devoted to prevent and remedy malfunctions and to warrant adequate behaviour. Testing, the subject of this paper, is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Indeed, software testing is a broad term encompassing a variety of activities along the development cycle and beyond, aimed at different goals. Hence, software testing research faces a collection of challenges. A consistent roadmap of the most relevant challenges to be addressed is here proposed. In it, the starting point is constituted by some important past achievements, while the destination consists of four identified goals to which research ultimately tends, but which remain as unreachable as dreams. The routes from the achievements to the dreams are paved by the outstanding research challenges, which are discussed in the paper along with interesting ongoing work.

REFERENCES

• 1: [1] L. Baresi and M. Young. Test oracles. Technical report, Dept. of Comp. and Information Science, Univ. of Oregon, 2001. http://www..cs.uoregon.edu/michal/pubs/oracles.html.
• 2: Emmanuel Bayse , Ana Cavalli , Manuel Núñez , Fatiha Zaïdi, A passive testing approach based on invariants: application to the WAP, Computer Networks and ISDN Systems, v.48 n.2, p.247-266, 6 June 205 [doi>10.1016/j.comnet.2004.09.009]
• 3: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 4: [4] A. Belinfante, L. Frantzen, and C. Schallhart. Tools for test case generation. In [21].
• 5: Stefan Berner , Roland Weber , Rudolf K. Keller, Observations and lessons learned from automated testing, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062556]
• 6: Gilles Bernot , Marie Claude Gaudel , Bruno Marre, Software testing based on formal specifications: a theory and a tool, Software Engineering Journal, v.6 n.6, p.387-405, Nov. 1991
• 7: Antonia Bertolino, ISSTA 2002 panel: is ISSTA research relevant to industrial users?, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 8: [8] A. Bertolino and E. Marchetti. Software testing (chapt.5). In P. Bourque and R. Dupuis, editors, Guide to the Software Engineering Body of Knowledge SWEBOK, 2004 Version, pages 5-1-5-16. IEEE Computer Society, 2004. http://www.swebok.org.
• 9: [9] A. Bertolino, E. Marchetti, and H. Muccini. Introducing a reasonably complete and coherent approach for model-based testing. Electr. Notes Theor. Comput. Sci., 116:85-97, 2005.
• 10: Antonia Bertolino , Andrea Polini, The Audition Framework for TestingWeb Services Interoperability, Proceedings of the 31st EUROMICRO Conference on Software Engineering and Advanced Applications, p.134-142, August 30-September 03, 2005 [doi>10.1109/EUROMICRO.2005.55]
• 11: [11] A. Bertolino, A. Polini, P. Inverardi, and H. Muccini. Towards anti-model-based testing. In Proc. DSN 2004 (Ext. abstract), pages 124-125, 2004.
• 12: Stefan Biffl , Aybüke Aurum , Barry Boehm , Hakan Erdogmus , Paul Grünbacher, Value-Based Software Engineering, Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 13: [13] S. Biffl, R. Ramler, and P. Gruenbacher. Value-based management of software testing. In [12].
• 14: Robert V. Binder, Testing object-oriented systems: models, patterns, and tools, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 15: Colin Blundell , Dimitra Giannakopoulou , Corina S. Pǎsǎreanu, Assume-guarantee testing, Proceedings of the 2005 conference on Specification and verification of component-based systems, September 05-06, 2005, Lisbon, Portugal
• 16: Gregor V. Bochmann , Alexandre Petrenko, Protocol testing: review of methods and relevance for software testing, Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis, p.109-124, August 17-19, 1994, Seattle, Washington, United States [doi>10.1145/186258.187153]
• 17: Marat Boshernitsan , Roongko Doong , Alberto Savoia, From daikon to agitator: lessons and challenges in building a commercial tool for developer testing, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146258]
• 18: Lionel C. Briand , Yvan Labiche , Yihong Wang, An Investigation of Graph-Based Class Integration Test Order Strategies, IEEE Transactions on Software Engineering, v.29 n.7, p.594-607, July 2003 [doi>10.1109/TSE.2003.1214324]
• 19: [19] L. Briand and A. Wolf, editors. Future of Software Engineering 2007. IEEE-CS Press, 2007.
• 20: Lionel C. Briand , Yvan Labiche , Michal M. Sówka, Automated, contract-based user testing of commercial-off-the-shelf components, Proceeding of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134300]
• 21: Manfred Broy , Bengt Jonsson , Joost-Pieter Katoen , Martin Leucker , Alexander Pretschner, Model-Based Testing of Reactive Systems: Advanced Lectures (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 22: Gerardo CanforaHarman , Massimiliano Di Penta, New Frontiers of Reverse Engineering, 2007 Future of Software Engineering, p.326-341, May 23-25, 2007 [doi>10.1109/FOSE.2007.15]
• 23: Gerardo Canfora , Massimiliano Di Penta, Testing Services and Service-Centric Systems: Challenges and Opportunities, IT Professional, v.8 n.2, p.10-17, March 2006 [doi>10.1109/MITP.2006.51]
• 24: Nelly Delgado , Ann Quiroz Gates , Steve Roach, A Taxonomy and Catalog of Runtime Software-Fault Monitoring Tools, IEEE Transactions on Software Engineering, v.30 n.12, p.859-872, December 2004 [doi>10.1109/TSE.2004.91]
• 25: [25] E. Dijkstra. Notes on structured programming. Technical Report 70-WSK03, Technological Univ. Eindhoven, 1970. http://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF.
• 26: Hyunsook Do , Sebastian Elbaum , Gregg Rothermel, Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact, Empirical Software Engineering, v.10 n.4, p.405-435, October 2005 [doi>10.1007/s10664-005-3861-2]
• 27: Matthew B. Dwyer , John Hatcliff , Robby Robby , Corina S. Pasareanu , Willem Visser, Formal Software Analysis Emerging Trends in Software Model Checking, 2007 Future of Software Engineering, p.120-136, May 23-25, 2007 [doi>10.1109/FOSE.2007.6]
• 28: Sebastian Elbaum , Hui Nee Chin , Matthew B. Dwyer , Jonathan Dokulil, Carving differential unit test cases from system test cases, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181806]
• 29: Sebastian Elbaum , Madeline Diep, Profiling Deployed Software: Assessing Strategies and Testing Opportunities, IEEE Transactions on Software Engineering, v.31 n.4, p.312-327, April 2005 [doi>10.1109/TSE.2005.50]
• 30: [30] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, to appear.
• 31: P. G. Frankl , E. J. Weyuker, Provable Improvements on Branch Testing, IEEE Transactions on Software Engineering, v.19 n.10, p.962-975, October 1993 [doi>10.1109/32.245738]
• 32: [32] L. Frantzen, J. Tretmans, and R. d. Vries. Towards model-based testing of web services. In Proc. Int. Workshop on Web Services - Modeling and Testing (WS-MaTe2006), pages 67- 82, 2006.
• 33: [33] L. Frantzen, J. Tretmans, and T. Willemse. A symbolic framework for model-based testing. In Proc. FATES/RV, LNCS 4262, pages 40-54. Springer-Verlag, 2006.
• 34: [34] M.-C. Gaudel. Formal methods and testing: Hypotheses, and correctness approximations. In Proc. FM 2005, LNCS 3582, pages 2-8. Springer-Verlag, 2005.
• 35: Patrice Godefroid , Nils Klarlund , Koushik Sen, DART: directed automated random testing, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 36: [36] J. B. Goodenough and S. L. Gerhart. Toward a theory of test data selection. IEEE Trans. Softw. Eng., 1(2):156-173, June 1975.
• 37: [37] R. Gotzhein and F. Khendek. Compositional testing of communication systems. In Proc. IFIP TestCom 2006, LNCS 3964, pages 227-244. Springer Verlag, May 2006.
• 38: [38] W. Grieskamp. Multi-paradigmatic model-based testing. In Proc. FATES/RV, pages 1-19. LNCS 4262, August 15-16, 2006.
• 39: Dick Hamlet, Subdomain testing of units and systems with state, Proceedings of the 2006 international symposium on Software testing and analysis, July 17-20, 2006, Portland, Maine, USA [doi>10.1145/1146238.1146249]
• 40: Dick Hamlet , Dave Mason , Denise Woit, Theory of software reliability based on components, Proceedings of the 23rd International Conference on Software Engineering, p.361-370, May 12-19, 2001, Toronto, Ontario, Canada
• 41: Dick Hamlet , Ross Taylor, Partition Testing Does Not Inspire Confidence (Program Testing), IEEE Transactions on Software Engineering, v.16 n.12, p.1402-1411, December 1990 [doi>10.1109/32.62448]
• 42: Mark Harman, The Current State and Future of Search Based Software Engineering, 2007 Future of Software Engineering, p.342-357, May 23-25, 2007 [doi>10.1109/FOSE.2007.29]
• 43: Mary Jean Harrold, Testing: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.61-72, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336532]
• 44: Bill Hetzel, The complete guide to software testing (2nd ed.), QED Information Sciences, Inc., Wellesley, MA, 1988
• 45: [45] W. Howden. Reliability of the path analysis testing strategy. IEEE Trans. Softw. Eng., SE-2(3):208- 215, 1976.
• 46: David Janzen , Hossein Saiedian, Test-Driven Development: Concepts, Taxonomy, and Future Direction, Computer, v.38 n.9, p.43-50, September 2005 [doi>10.1109/MC.2005.314]
• 47: [47] JUnit.org. http://www.junit.org/index.htm.
• 48: Natalia Juristo , Ana M. Moreno , Sira Vegas, Reviewing 25 Years of Testing Technique Experiments, Empirical Software Engineering, v.9 n.1-2, p.7-44, March 2004 [doi>10.1023/B:EMSE.0000013513.48963.1b]
• 49: Gregory M. Kapfhammer , Mary Lou Soffa , Daniel Mosse, Testing in resource constrained execution environments, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101984]
• 50: [50] C. Keum, S. Kang, I.-Y. Ko, J. Baik, and Y.-I. Choi. Generating test cases for web services using extended finite state machine. In Proc. IFIP TestCom 2006, LNCS 3964, pages 103-117. Springer Verlag, 2006.
• 51: Kim G. Larsen , Marius Mikucionis , Brian Nielsen , Arne Skou, Testing real-time embedded software using UPPAAL-TRON: an industrial case study, Proceedings of the 5th ACM international conference on Embedded software, September 18-22, 2005, Jersey City, NJ, USA [doi>10.1145/1086228.1086283]
• 52: Yves Le Traon , Benoit Baudry , Jean-Marc Jezequel, Design by Contract to Improve Software Vigilance, IEEE Transactions on Software Engineering, v.32 n.8, p.571-586, August 2006 [doi>10.1109/TSE.2006.79]
• 53: Gareth Lee , John Morris , Kris Parker , Gary A. Bundell , Peng Lam, Using symbolic execution to guide test generation: Research Articles, Software Testing, Verification & Reliability, v.15 n.1, p.41-61, March 2005 [doi>10.1002/stvr.v15:1]
• 54: Timothy C. Lethbridge , Jorge Diaz-Herrera , Richard J. Jr. LeBlanc , J. Barrie Thompson, Improving software practice through education: Challenges and future trends, 2007 Future of Software Engineering, p.12-28, May 23-25, 2007 [doi>10.1109/FOSE.2007.13]
• 55: Zhongjie Li , Wei Sun , Zhong Bo Jiang , Xin Zhang, BPEL4WS Unit Testing: Framework and Implementation, Proceedings of the IEEE International Conference on Web Services (ICWS'05), p.103-110, July 11-15, 2005 [doi>10.1109/ICWS.2005.31]
• 56: Ben Liblit , Alex Aiken , Alice X. Zheng , Michael I. Jordan, Bug isolation via remote program sampling, Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation, June 09-11, 2003, San Diego, California, USA
• 57: Michael R. Lyu, Software Reliability Engineering: A Roadmap, 2007 Future of Software Engineering, p.153-170, May 23-25, 2007 [doi>10.1109/FOSE.2007.24]
• 58: Michael R. Lyu, Handbook of software reliability and system reliability, McGraw-Hill, Inc., Hightstown, NJ, 1996
• 59: [59] L. Mariani and M. Pezzè. Dynamic detection of COTS components incompatibility. IEEE Software, to appear.
• 60: Phil McMinn, Search-based software test data generation: a survey: Research Articles, Software Testing, Verification & Reliability, v.14 n.2, p.105-156, June 2004 [doi>10.1002/stvr.v14:2]
• 61: [61] Microsoft Research. Customer experience improvement program, 2006. http://www.microsoft.com/ products/ceip/.
• 62: [62] E. F. Moore. Gedanken-experiments on sequential machines. Automata Studies, pages 129-153, 1956.
• 63: [63] NIST. The economic impacts of inadequate infrastructure for software testing, May 2002. http://www.nist.gov/director/prog-ofc/report02-3.pdf.
• 64: Alessandro Orso , Taweesup Apiwattanapong , Mary Jean Harrold, Leveraging field data for impact analysis and regression testing, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 65: Alessandro Orso , Bryan Kennedy, Selective capture and replay of program executions, Proceedings of the third international workshop on Dynamic analysis, p.1-7, May 17-17, 2005, St. Louis, Missouri
• 66: Thomas J. Ostrand , Elaine J. Weyuker , Robert M. Bell, Predicting the Location and Number of Faults in Large Software Systems, IEEE Transactions on Software Engineering, v.31 n.4, p.340-355, April 2005 [doi>10.1109/TSE.2005.49]
• 67: [67] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. Technical Report MSR-TR-2006-125, Microsoft Research, Redmond, WA.
• 68: Larry Peterson , Tom Anderson , David Culler , Timothy Roscoe, A blueprint for introducing disruptive technology into the Internet, ACM SIGCOMM Computer Communication Review, v.33 n.1, p.59-64, January 2003 [doi>10.1145/774763.774772]
• 69: J. H. Poore , Harlan D. Mills , David Mutchler, Planning and Certifying Software System Reliability, IEEE Software, v.10 n.1, p.88-99, January 1993 [doi>10.1109/52.207234]
• 70: [70] M. J. Rehman, F. Jabeen, A. Bertolino, and A. Polini. Testing software components for integration: a survey of issues and techniques. Software Testing, Verification and Reliability , to appear.
• 71: Arthur Alexander Reyes , Debra Richardson, Siddhartha: A Method for Developing Domain-Specific Test Driver Generators, Proceedings of the 14th IEEE international conference on Automated software engineering, p.81, October 12-15, 1999
• 72: Matthew J. Rutherford , Antonio Carzaniga , Alexander L. Wolf, Simulation-based test adequacy criteria for distributed systems, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181804]
• 73: David Saff , Shay Artzi , Jeff H. Perkins , Michael D. Ernst, Automatic test factoring for java, Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, November 07-11, 2005, Long Beach, CA, USA [doi>10.1145/1101908.1101927]
• 74: David Saff , Michael D. Ernst, An experimental evaluation of continuous testing during development, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 75: Koushik Sen , Darko Marinov , Gul Agha, CUTE: a concolic unit testing engine for C, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, September 05-09, 2005, Lisbon, Portugal
• 76: Avik Sinha , Carol Smidts, HOTTest: A model-based test design technique for enhanced testing of domain-specific applications, ACM Transactions on Software Engineering and Methodology (TOSEM), v.15 n.3, p.242-278, July 2006 [doi>10.1145/1151695.1151697]
• 77: Dag I. K. Sjoberg , Tore Dyba , Magne Jorgensen, The Future of Empirical Methods in Software Engineering Research, 2007 Future of Software Engineering, p.358-378, May 23-25, 2007 [doi>10.1109/FOSE.2007.30]
• 78: Nikolai Tillmann , Wolfram Schulte, Unit Tests Reloaded: Parameterized Unit Testing with Symbolic Execution, IEEE Software, v.23 n.4, p.38-47, July 2006 [doi>10.1109/MS.2006.117]
• 79: [79] J. Tretmans. Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools, 17:103- 120, 1996.
• 80: Mark Utting , Bruno Legeard, Practical Model-Based Testing: A Tools Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2006
• 81: [81] M. van der Bijl, A. Rensink, and J. Tretmans. Compositional testing with ioco. In Proc. FATES 2003, LNCS 2931, 2003.
• 82: [82] S. Vegas, N. Juristo, and V. Basili. Packaging experiences for improving testing technique selection. The Journal of Systems and Software, 79(11):1606-1618, Nov. 2006.
• 83: Joachim Wegener , Matthias Grochtmann, Verifying Timing Constraints of Real-Time Systems by Meansof Evolutionary Testing, Real-Time Systems, v.15 n.3, p.275-298, Nov. 1998 [doi>10.1023/A:1008096431840]
• 84: Elaine J. Weyuker , Filippos I. Vokolos, Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study, IEEE Transactions on Software Engineering, v.26 n.12, p.1147-1156, December 2000 [doi>10.1109/32.888628]
• 85: [85] E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465-470, 1982.
• 86: Murray Woodside , Greg Franks , Dorina C. Petriu, The Future of Software Performance Engineering, 2007 Future of Software Engineering, p.171-187, May 23-25, 2007 [doi>10.1109/FOSE.2007.32]
• 87: Arvind S. Krishna , Cemal Yilmaz , Atif M. Memon , Adam A. Porter , Douglas C. Schmidt , Aniruddha Gokhale , Balachandran Natarajan, Preserving Distributed Systems' Critical Properties: A Model-Driven Approach, IEEE Software, v.21 n.6, p.32-40, November 2004 [doi>10.1109/MS.2004.50]
• 88: Hong Zhu, A Formal Analysis of the Subsume Relation Between Software Test Adequacy Criteria, IEEE Transactions on Software Engineering, v.22 n.4, p.248-255, April 1996 [doi>10.1109/32.491648]
• 89: Hong Zhu , Patrick A. V. Hall , John H. R. May, Software unit test coverage and adequacy, ACM Computing Surveys (CSUR), v.29 n.4, p.366-427, Dec. 1997 [doi>10.1145/267580.267590]
• 90: Hong Zhu , Xudong He, A Theory of Behaviour Observation in Software Testing, Oxford Brookes University, 1999

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.4 Software/Program Verification

  F. Theory of Computation
  F.3 LOGICS AND MEANINGS OF PROGRAMS
      F.3.2 Semantics of Programming Languages
          Subjects: Program analysis

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.2 Automatic Programming
          Subjects: Program verification

General Terms:
Theory, Verification