Breeding swarms: a new approach to recurrent neural network training

ABSTRACT

This paper shows that a novel hybrid algorithm, Breeding Swarms, performs equal to, or better than, Genetic Algorithms and Particle Swarm Optimizers when training recurrent neural networks. The algorithm was found to be robust and scale well to very large networks, ultimately outperforming Genetic Algorithms and Particle Swarm Optimization in 79 of 80 tested networks. This research shows that the Breeding Swarm algorithm is a viable option when choosing an algorithm to train recurrent neural networks.

REFERENCES

• 1: Peter J. Angeline, Evolutionary Optimization Versus Particle Swarm Optimization: Philosophy and Performance Differences, Proceedings of the 7th International Conference on Evolutionary Programming VII, p.601-610, March 25-27, 1998
• 2: P. Angeline, G. Saunders, and J. Pollack. An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1):54--65, January 1994.
• 3: J. Dowling. Neurons and Networks: An Introduction to Neuroscience. The Belknap Press of Harvard University Press, Cambridge, MA, 1992.
• 4: Russell C. Eberhart , Yuhui Shi, Comparison between Genetic Algorithms and Particle Swarm Optimization, Proceedings of the 7th International Conference on Evolutionary Programming VII, p.611-616, March 25-27, 1998
• 5: M. El Choubassi, H. El Khoury, C. Jabra Alagha, J. Skaf, and M. Al-Alaoui. Arabic speech recognition using recurrent neural networks. In IEEE International Symposium on Signal Processing and Information Technology (ISSPIT 2003), Darmstadt, Germany, December 14--17 2003.
• 6: L. Eshelman and J. Schaffer. Real-coded genetic algorithms and interval-schemata. In L. D. Whitley, editor, Foundations of Genetic Algorithms 2, pages 187--202. Morgan Kaufmann, San Mateo, CA, 1993.
• 7: A. Esparcia-Alcazar and K. Sharman. Evolving recurrent neural network architectures by genetic programming. In J. R. Koza and et. al., editors, Genetic Programming 1997: Proceedings of the Second Annual Conference, pages 89--94, Stanford University, CA, USA, 13-16 1997. Morgan Kaufmann.
• 8: C. Lee Giles , Steve Lawrence , Ah Chung Tsoi, Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference, Machine Learning, v.44 n.1-2, p.161-183, July-August 2001 [doi>10.1023/A:1010884214864]
• 9: Barbara Hammer, Learning with Recurrent Neural Networks, Springer-Verlag New York, Inc., Secaucus, NJ, 2000
• 10: John H. Holland, Adaptation in natural and artificial systems, MIT Press, Cambridge, MA, 1992
• 11: B. Horne and C. Giles. An experimental comparison of recurrent neural networks. In G. Tesauro and et. al., editors, In Advances in Neural Information Processing Systems, volume 7, pages 697--704. The MIT Press, 1995.
• 12: E. Kandel and J. Schwartz. Principles of Neuroscience, 2nd Edition. Elsevier, New York, NY, 1985.
• 13: James Kennedy , Russell C. Eberhart, Swarm intelligence, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001
• 14: M. Mandischer. Evolving recurrent neural networks with non-binary encoding". In In Proc. Second IEEE Intl. Conf. Evelutionary Computation (ICEC '95), volume 2, pages 584--589, Perth, Australia, 1995. IEEE Press, Piscataway, NJ.
• 15: K. Ohya. A sound synthesis by recurrent neural network. In E. Michie, editor, Proceedings of the 1995 International Computer Music Conference, pages 420--423, San Francisco: International Computer Music Association, 1995.
• 16: G. Saunders, P. Angeline, and J. Pollack. Structural and behavioral evolution of recurrent networks. In J. Cowan and et. al., editors, Advances in Neural Information Processing Systems, volume 6, pages 88--95. Morgan Kaufmann Publishers, Inc., 1994.
• 17: M. Settles, B. Rodebaugh, and T. Soule. Comparison of genetic algorithm and particle swarm optimizer when evolving a recurrent neural network. In E. Cantú-Paz and et. al., editors, Genetic and Evolutionary Computation -- GECCO-2003, volume 2723 of LNCS, pages 148--149, Chicago, 12-16 July 2003. Springer-Verlag.
• 18: G. Shepherd. Neurobiology. Oxford University Press, New York, NY, 1994.
• 19: T. Soule, Y. Chen, and R. Wells. Evolving a strongly recurrent neural network to simulate biological neurons. In In the proceedings of The 28th Annual Conference of the IEEE Industrial Electronics Society, 2002.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.8 Problem Solving, Control Methods, and Search
          Subjects: Heuristic methods

Additional Classification:
  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.1 Models of Computation
          Subjects: Self-modifying machines (e.g., neural networks)

General Terms:
Algorithms

Keywords:
breeding swarms, genetic algorithms, particle swarm optimization, recurrent neural networks