Evolutionary testing of autonomous software agents

ABSTRACT

A system built in terms of autonomous agents may require even greater correctness assurance than one which is merely reacting to the immediate control of its users. Agents make substantial decisions for themselves, so thorough testing is an important consideration. However, autonomy also makes testing harder; by their nature, autonomous agents may react in different ways to the same inputs over time, because, for instance they have changeable goals and knowledge. For this reason, we argue that testing of autonomous agents requires a procedure that caters for a wide range of test case contexts, and that can search for the most demanding of these test cases, even when they are not apparent to the agents' developers. In this paper, we address this problem, introducing and evaluating an approach to testing autonomous agents that uses evolutionary optimization to generate demanding test cases.

REFERENCES

• 1: J. A. Botía, A. López-Acosta, and A. F. Gómez-Skarmeta. ACLAnalyser: A tool for debugging multi-agent systems. In Proc. of the European Conference on Artificial Intelligence, pages 967--968, 2004.
• 2: M. E. Bratman. Intentions, Plans and Practical Reason. Harvard University Press, 1987.
• 3: Paolo Bresciani , Anna Perini , Paolo Giorgini , Fausto Giunchiglia , John Mylopoulos, Tropos: An Agent-Oriented Software Development Methodology, Autonomous Agents and Multi-Agent Systems, v.8 n.3, p.203-236, May 2004 [doi>10.1023/B:AGNT.0000018806.20944.ef]
• 4: Oliver Bühler , Joachim Wegener, Evolutionary functional testing, Computers and Operations Research, v.35 n.10, p.3144-3160, October, 2008 [doi>10.1016/j.cor.2007.01.015]
• 5: Roberta Coelho , Uirá Kulesza , Arndt von Staa , Carlos Lucena, Unit testing in multi-agent systems using mock agents and aspects, Proceedings of the 2006 international workshop on Software engineering for large-scale multi-agent systems, May 22-23, 2006, Shanghai, China [doi>10.1145/1138063.1138079]
• 6: Anne Dardenne , Axel van Lamsweerde , Stephen Fickas, Goal-directed requirements acquisition, Science of Computer Programming, v.20 n.1-2, p.3-50, April 1993
• 7: Oguz Dikenelli , Riza Cenk Erdur , Ozgur Gumus, SEAGENT: a platform for developing semantic web based multi agent systems, Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, July 25-29, 2005, The Netherlands [doi>10.1145/1082473.1082728]
• 8: E. Gamma and K. Beck. JUnit: A Regression Testing Framework. http://www.junit.org, 2000.
• 9: M. P. Georgeff and F. F. Ingrand. Decision-making in an embedded reasoning system. In Proc. of the International Joint Conferences on Artificial Intelligence, pages 972--978, 1989.
• 10: Mark Harman, The Current State and Future of Search Based Software Engineering, 2007 Future of Software Engineering, p.342-357, May 23-25, 2007 [doi>10.1109/FOSE.2007.29]
• 11: Henry Hexmoor , Cristiano Castelfranchi , Rino Falcone, Agent Autonomy, Kluwer Academic Publishers, Norwell, MA, 2003
• 12: D. N. Lam and K. S. Barber. Programming Multi-Agent Systems, chapter Debugging Agent Behavior in an Implemented Agent System, pages 104--125. Springer Berlin / Heidelberg, 2005.
• 13: P. McMinn and M. Holcombe. The state problem for evolutionary testing. In Proc. of the Genetic and Evolutionary Computation Conference, pages 2488--2498. Springer-Verlag, 2003.
• 14: C. D. Nguyen, A. Perini, and P. Tonella. Automated continuous testing of multi-agent systems. In The fifth European Workshop on Multi-Agent Systems, December 2007.
• 15: Manuel Núñez , Ismael Rodríguez , Fernando Rubio, Specification and testing of autonomous agents in e-commerce systems: Research Articles, Software Testing, Verification & Reliability, v.15 n.4, p.211-233, December 2005 [doi>10.1002/stvr.v15:4]
• 16: A. Pokahr, L. Braubach, and W. Lamersdorf. Jadex: A BDI Reasoning Engine, chapter Multi-Agent Programming. Kluwer Book, 2005.
• 17: L. F. Rodrigues, G. R. de Carvalho, R. de Barros Paes, and C. J. P. de Lucena. Towards an Integration Test Architecture for Open MAS. In Proc. of the 1st Workshop on Software Engineering for Agent-oriented Systems / SBES, 2005.
• 18: TILAB. Java agent development framework. http://jade.tilab.com/.
• 19: A. M. Tiryaki, S. Öztuna, O. Dikenelli, and R. C. Erdur. Sunit: A unit testing framework for test driven development of multi-agent systems. In Proc. of the 7th International Workshop on Agent-Oriented Software Engineering, pages 156--173, 2006.
• 20: Frank Vavak , Terence C. Fogarty, A Comparative Study of Steady State and Generational Genetic Algorithms, Selected Papers from AISB Workshop on Evolutionary Computing, p.297-304, April 01-02, 1996
• 21: J. Wegener. Stochastic Algorithms: Foundations and Applications, chapter Evolutionary Testing Techniques, pages 82--94. Springer Berlin / Heidelberg, 2005.
• 22: Danny Weyns , Andrea Omicini , James Odell, Environment as a first class abstraction in multiagent systems, Autonomous Agents and Multi-Agent Systems, v.14 n.1, p.5-30, February 2007 [doi>10.1007/s10458-006-0012-0]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

General Terms:
Reliability, Verification

Keywords:
autonomous agents, evolutionary testing