Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Despite the popularity and success of neural networks in research, the number of resulting commercial or industrial applications have been limited. A primary cause of this lack of adoption is due to the fact that neural networks are usually implemented as software running on general-purpose processors. Algorithms to implement a neural network in software are typically O(n²) problems -- as a result, neural networks are unable to provide the performance and scalability required in non-academic settings.

In this paper, we investigate how FPGAs can be used to take advantage of the inherent parallelism in neural networks to provide a better implementation in terms of scalability and performance. We will focus on the Restricted Boltzmann machine, a popular type of neural network, because its architecture is particularly well-suited to hardware designs. The proposed, multi-purpose hardware framework is designed to reduce the O(n²2) problem into an O(n) implementation while only requiring O(n) resources. The framework is tested on a Xilinx Virtex II-Pro XC2VP70 FPGA running at 100MHz. The resources support a Restricted Boltzmann machine of 128x128 nodes, which results in a computational speed of 1.02 billion connection-updates-per-second and a speed-up of 35 fold over an optimized C program running on a 2.8GHz Intel processor.

REFERENCES

• 1: Geoffrey E. Hinton , Simon Osindero , Yee-Whye Teh, A fast learning algorithm for deep belief nets, Neural Computation, v.18 n.7, p.1527-1554, July 2006 [doi>10.1162/neco.2006.18.7.1527]
• 2: G. E. Hinton and R. R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," Science, vol. 313, pp. 504--507, July 2006.
• 3: C. S. Lindsey and T. Lindblad, "Survey of neural network hardware," Applications and Science of Artificial Neural Networks, pp. 1194--1205, 1995.
• 4: Y. Liao, "Neural Networks in Hardware: A Survey," tech. rep., Santa Cruz, CA, USA, 2001.
• 5: J. Zhu and P. Sutton, "FPGA Implementations of Neural Networks -- A Survey of a Decade of Progress," Lecture Notes in Computer Science, no. 2778, pp. 1062--1066, 2003.
• 6: Pedro Ferreira , Pedro Ribeiro , Ana Antunes , Fernando Morgado Dias, A high bit resolution FPGA implementation of a FNN with a new algorithm for the activation function, Neurocomputing, v.71 n.1-3, p.71-77, December, 2007 [doi>10.1016/j.neucom.2006.11.028]
• 7: D. Shen, L. Jin, and X. Ma, "FPGA Implementation of Feature Extraction and Neural Network Classifier for Handwritten Digit Recognition," Lecture notes in computer science, vol. 3173, pp. 988--995, 2004.
• 8: P. Smolensky, Information processing in dynamical systems: foundations of harmony theory, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 9: Y. Freund and D. Haussler, "Unsupervised Learning of Distributions on Binary Vectors Using Two Layer Networks," NIPS, pp. 912--919, 1992.
• 10: D. Geman and S. Geman, "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 6, no. 6, pp. 721--741, 1984.
• 11: D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, "A Learning Algorithm for Boltzmann Machines," Cognitive Science, vol. 9, pp. 147--169, 1985.
• 12: G. E. Hinton , T. J. Sejnowski, Learning and relearning in Boltzmann machines, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 13: M. Saldana and P. Chow, "TMD-MPI: An MPI Implementation for Multiple Processors across Multiple FPGAs," IEEE International Conference on Field-Programmable Logic and Applications (FPL 2006), pp. 329--334, 2006.
• 14: M. A. Carreira-Perpinan and G. E. Hinton, "On Contrastive Divergence Learning," Artificial Intelligence and Statistics, 2005.
• 15: Pedro Ferreira , Pedro Ribeiro , Ana Antunes , Fernando Morgado Dias, A high bit resolution FPGA implementation of a FNN with a new algorithm for the activation function, Neurocomputing, v.71 n.1-3, p.71-77, December, 2007 [doi>10.1016/j.neucom.2006.11.028]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.3 SPECIAL-PURPOSE AND APPLICATION-BASED SYSTEMS

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.5 Implementation

General Terms:
Design, Performance

Keywords:
complexity reduction, fpga, high-performance computing, neural network hardware, restricted boltzmann machines, scalable hardware designs