Connectionist ideas and algorithms

ABSTRACT

In our quest to build intelligent machines, we have but one naturally occurring model: the human brain. It follows that one natural idea for artificial intelligence (AI) is to simulate the functioning of the brain directly on a computer. Indeed, the idea of building an intelligent machine out of artificial neurons has been around for quite some time. Some early results on brain-line mechanisms were achieved by [18], and other researchers pursued this notion through the next two decades, e.g., [1, 4, 19, 21, 24]. Research in neural networks came to a virtual halt in the 1970s, however, when the networks under study were shown to be very weak computationally. Recently, there has been a resurgence of interest in neural networks. There are several reasons for this, including the appearance of faster digital computers on which to simulate larger networks, interest in building massively parallel computers, and most importantly, the discovery of powerful network learning algorithms. The new neural network architectures have been dubbed connectionist architectures. For the most part, these architectures are not meant to duplicate the operation of the human brain, but rather receive inspiration from known facts about how the brain works. They are characterized by Large numbers of very simple neuron-like processing elements; Large numbers of weighted connections between the elements—the weights on the connections encode the knowledge of a network; Highly parallel, distributed control; and Emphasis on learning internal representations automatically. Connectionist researchers conjecture that thinking about computation in terms of the brain metaphor rather than the digital computer metaphor will lead to insights into the nature of intelligent behavior. Computers are capable of amazing feats. They can effortlessly store vast quantities of information. Their circuits operate in nanoseconds. They can perform extensive arithmetic calculations without error. Humans cannot approach these capabilities. On the other hand, humans routinely perform simple tasks such as walking, talking, and commonsense reasoning. Current AI systems cannot do any of these things better than humans. Why not? Perhaps the structure of the brain is somehow suited to these tasks, and not suited to tasks like high-speed arithmetic calculation. Working under constraints suggested by the brain may make traditional computation more difficult, but it may lead to solutions to AI problems that would otherwise be overlooked. What constraints, then, does the brain offer us? First of all, individual neurons are extremely slow devices when compared to their counterparts in digital computers. Neurons operate in the millisecond range, an eternity to a VLSI designer. Yet, humans can perform extremely complex tasks, like interpreting a visual scene or understanding a sentence, in just a tenth of a second. In other words, we do in about a hundred steps what current computers cannot do in ten million steps. How can this be possible? Unlike a conventional computer, the brain contains a huge number of processing elements that act in parallel. This suggests that in our search for solutions, we look for massively parallel algorithms that require no more than 100 processing steps [9]. Also, neurons are failure-prone devices. They are constantly dying (you have certainly lost a few since you began reading this article), and their firing patterns are irregular. Components in digital computers, on the other hand, must operate perfectly. Why? Such components store bits of information that are available nowhere else in the computer: the failure of one component means a loss of information. Suppose that we built AI programs that were not sensitive to the failure of a few components, perhaps by using redundancy and distributing information across a wide range of components? This would open the possibility of very large-scale implementations. With current technology, it is far easier to build a billion-component integrated circuit in which 95 percent of the components work correctly than it is to build a perfectly functioning million-component machine [8]. Another thing people seem to be able to do better than computers is handle fuzzy situations. We have very large memories of visual, auditory, and problem-solving episodes, and one key operation in solving new problems is finding closest matches to old situations. Inexact matching is something brain-style models seem to be good at, because of the diffuse and fluid way in which knowledge is represented. The idea behind connectionism, then, is that we may see significant advances in AI if we approach problems from the point of view of brain-style computation rather than rule-based symbol manipulation. At the end of this article, we will look more closely at the relationship between connectionist and symbolic AI.

REFERENCES

• 1: Ashby, W. R. Design for a Brain. Wiley, New York. 1952.
• 2: Dana H. Ballard, Parameter Nets, Artificial Intelligence, v.22 n.3, p.235-267, April 1984 [doi>10.1016/0004-3702(84)90052-3]
• 3: Barto, A. G. Learning by stalistical cooperation of self-interested teuron-like comptuting elements. Human Neurobiology 4,4 (1985).
• 4: Block, H. D. The perceptron' A model for brain functiolling. Rez,. Mod. phy. 34, I (1962), 123-135. _
• 5: Cottrell, G. W., Munro P., and Zipset D. learning internal representations from gray-scale images' An example of cxtensional programmiilg. In Proceedings ' the Ni,th Annual ConFrence of the Cognitive Science Society (1987), pp. 461-473.
• 6: Mark Derthick , Geoffrey E. Hinton, Mundane reasoning by parallel constraint satisfaction, 1988
• 7: Fahlman, S. E. Faster-learning variations on back-propagation: An empirical studv, in Proceedings q'the 1988 Connecliouis.' Models Summer School. Morgan Katlfmann Publishers, San Mateo, Calif. 1988, pp. 38- 51,
• 8: Scott F. Fahlman , Geoffrey E. Hinton, Connectionist architectures for artificial intelligence, Computer, v.20 n.1, p.100-109, Jan. 1987 [doi>10.1109/MC.1987.1663364]
• 9: Feldman, J. A. and Ballard, D. H. Connectionist models and their properties. Cogn. Sci. 6, 3 (1985), 205-254.
• 10: Gorlnan, R. and Sejnowski, 'F. J. Analysis of hidden units in a layered network to classify sonar targets. Neural Networks I, 1 (1988), 75-89,
• 11: G. E. Hinton , T. J. Sejnowski, Learning and relearning in Boltzmann machines, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 12: Hopficld, J..}. Neural networks and physical systems with emergent collective cotnptttational abilities, in proceedings of the National Acodamy of Sciences USA 79, 8 (1982), pp. 2554- 2558.
• 13: Hopfield, .J.J. and Tank, D. W. Neural' computalion of decisions in optimizatit)n problems. Biol. Cybern. 52, 3 (1985), 141-152.
• 14: jordatl, M. I. Supervised learning and systems with excess degrees of freedom. In Proceedings of the 1988 Connectiouist Models Summer School. Morgan Kimflnann Publishers, San Mateco, Calif., 1988, pp. 62-75.
• 15: Kirkpatrick, S., Gelatt, Jr., C. D., and Vecchi, M. P Optimization bv simulated allnealing. Science 220, 4598 (1983).
• 16: LeCun, Y. Une procedure d'apprmltissage pour reseau a seauil assvmetrique (a learning procedure tot asvmmetric threshold networks). In Proceedings of Cognitiva 85, (paris, 1985), pp, 599-604.
• 17: Richard P. Lippmann, Review of neural networks for speech recognition, Neural Computation, v.1 n.1, p.1-38, Spring 1990 [doi>10.1162/neco.1989.1.1.1]
• 18: McCulloch, W. S. and Pitts, W. A h)gical calculus of the ideas immanent in neural nets. Bulletin o/ Math. Biophys. 5 (1943), 115-137.
• 19: Minsky. M. Neural Nets a,d the Brai,-Model Problem. Ph.D dissertation. Princeton Lrniversity, Princeton, New Jersey, 1954.
• 20: Marvin Minsky , Seymour Papert, Perceptrons, Neurocomputing: foundations of research, MIT Press, Cambridge, MA, 1988
• 21: Minskv, M. and Sell'ridge, O. G, LearniJlg in neural nets. In Proceedings of the Fourth London Symposium on Information Theory (August 29 to September 2, London), Academic Press, New York 1961.
• 22: Parker, D, B, Leamting Logic. 'l'ech. Rep. 'FR-47, M IT Center for Computational Research in Economics and Managclnent Science, 1985,
• 23: Dean A. Pomerleau, ALVINN: an autonomous land vehicle in a neural network, Advances in neural information processing systems 1, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1989
• 24: Rosenblatt, F. Principles of Neurodrnamics: Perceptrons and the Theory of Brain Mechanisms. Spartan Books, Washington, D.C. 1962.
• 25: D. E. Rumelhart , G. E. Hinton , R. J. Williams, Learning internal representations by error propagation, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 26: D. E. Rumelhart , J. L. McClelland, On learning the past tenses of English verbs, Parallel distributed processing: explorations in the microstructure of cognition, vol. 2: psychological and biological models, MIT Press, Cambridge, MA, 1986
• 27: D. E. Rumelhart , D. Zipser, Feature discovery by competitive learning, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986
• 28: Sejnowski, T. J. and Rosenberg, C. R. Parallel networks that learn to pronounce English text. Complex Srstems 1, 1 (1987), 145-168.
• 29: G. Tesauro , J. J. Sejnowski, A parallel network that learns to play backgammon, Artificial Intelligence, v.39 n.3, p.357-390, July 1989 [doi>10.1016/0004-3702(89)90017-9]
• 30: Touretzkv, D. and Hinton, G. E. A distributed connectionist production system. Cog. Sci. 12, 3 (1988), 423-466,
• 31: Werbos, P.J. Beyond Regression: New Tools t'(}r Prediction and Analysis in the Behavioral Sciences. Ph.D. dissertation, Harvard University, Cambridge, Mass., 1974.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Connectionism and neural nets

General Terms:
Algorithms, Design