A matrix calculus for neural nets: II

In a previous paper a method was given by which the efferent activity of an idealized neural net could be calculated from a given afferent pattern. Those results are extended in the present paper. Conditions are given under which nets may be considered equivalent. Rules are given for the reduction or extension of a net to an equivalent net. A procedure is given for constructing a net which has the property of converting each of a given set of afferent activity patterns into its corresponding prescribed efferent activity pattern.     

Self-control in neural nets

This paper deals with the problem of self-controlling nets of Caianiello's type. These nets control themselves through their own elements.     

Memory topology of neural nets

Aiming at a topology of memory-growth, a computer-simulated mechanism is developed here, based on the probabilistic neural model previously developed by Anninos, Csermeley & Harth.The essential topology of this mechanism reflects the generally accepted idea that memory growth is achieved by means of association processes, i.e. through the cross-referencing of new inputs with already acquired information. In terms of computer-simulation such mechanism is developed by following the formalism of Set Theory. So the memory units, i.e. the neural netlets are equated to subsets of a set which represents a larger memory-system. Consistently with this formalism, association is a necessary, although non-sufficient condition for the growth of memory and the Boolean overlap of two subsets, i.e. of two netlets, is identified with the cross-reference mechanism which we assume to be the basis of memory growth.     

On matrices of neural nets

The structure of a complete or incomplete neural net is represented here by several matrices. The activity equation of the net follows in a general form. A chain or cycle is defined as a neural structure whose connection matrix is unitary. We can compute the number of simple chains by a recurrent formula. Academia Sinica     

Schema generation in recurrent neural nets for intercepting a moving target

The grasping of a moving object requires the development of a motor strategy to anticipate the trajectory of the target and to compute an optimal course of interception. During the performance of perception-action cycles, a preprogrammed prototypical movement trajectory, a motor schema, may highly reduce the control load. Subjects were asked to hit a target that was moving along a circular path by means of a cursor. Randomized initial target positions and velocities were detected in the periphery of the eyes, resulting in a saccade toward the target. Even when the target disappeared, the eyes followed the target’s anticipated course. The Gestalt of the trajectories was dependent on target velocity. The prediction capability of the motor schema was investigated by varying the visibility range of cursor and target. Motor schemata were determined to be of limited precision, and therefore visual feedback was continuously required to intercept the moving target. To intercept a target, the motor schema caused the hand to aim ahead and to adapt to the target trajectory. The control of cursor velocity determined the point of interception. From a modeling point of view, a neural network was developed that allowed the implementation of a motor schema interacting with feedback control in an iterative manner. The neural net of the Wilson type consists of an excitation-diffusion layer allowing the generation of a moving bubble. This activation bubble runs down an eye-centered motor schema and causes a planar arm model to move toward the target. A bubble provides local integration and straightening of the trajectory during repetitive moves. The schema adapts to task demands by learning and serves as forward controller. On the basis of these model considerations the principal problem of embedding motor schemata in generalized control strategies is discussed.     

A diffusion model of a neuron and neural nets

In the paper a diffusion model of a neuron is treated. A new, less restrictive than usually, condition of applicability of a diffusion model is presented. As a result the point-process-to-point-process model of a neuron is obtained, which produces an output signal of the same kind as the accepted input signals.     

A note on neural nets

A neural mechanism is described which provides for a perception which is invariant with respect to movement of the stimulus.     

A statistical consequence of the logical calculus of nervous nets

A formal method is derived for converting logical relations among the actions of neurons in a net into statistical relations among the frequencies of their impulses.     

Symmetry breaking in neural nets

In this paper two well-known homogeneous models of neural nets undergoing symmetry-breaking transitions are studied in order to see if, after the transition, there is the appearance of Goldstone models. These have been found only in an approximate way; there are indications, however, that they can play a prominent role when the tissue is subjected to external inputs, constraining it to be slaved to the characteristics of those. This circumstance should be essential in explaining how a structured net can store complex inputs and give subsequently ordered outputs.     

A benchmark for how well neural nets generalize

This paper discusses the problem of extending the domain of learning sets and introduces HERBIE, a program which achieves this through graphical procedures rather than via neural networks. It is argued that for theoretical reasons HERBIE is well-suited to serving as a benchmark for measuring generalization efficacy, and therefore to serving as a means of testing claims of emergent distributed intelligence in neural nets. The successful results of tests of HERBIE as a pattern recognizer are presented, and HERBIE's behavior is favorably compared to neural nets for several real generalization problems. Finally, applications of HERBIE independent of its serving as a generalization benchmark, particularly in the area of cognitive science, are discussed.     

A canonical form for neural nets without circles

By “neural net” will be meant “neural net without circles.” Every neural net effects a transformation from inputs (i.e., firing patterns of the input neurons) to outputs (firing patterns of the output neurons). Two neural nets will be calledequivalent if they effect the same transformation from inputs to outputs. A canonical form is found for neural nets with respect to equivalence; i.e., a class of neural nets is defined, no two of which are equivalent, and which contains a neural net equivalent to any given neural net. This research was supported by the U.S. Air Force under Contract AF 49(638)-414 monitored by the Air Force Office of Scientific Research.     

Methods of analysis of neural nets

The properties of isotropy, smoothness, minimum curvature and locality suggest the shape of filled-in contours between two boundary edges. The contours are composed of the arcs of two circles tangent to the given edges, meeting smoothly, and minimizing the total curvature. It is shown that shapes meeting all the above requirement can be generated by a network which performs simple, local computations. It is suggested that the filling-in process plays an important role in the early processing of visual information.     

The wiring-in of neural nets revisited

Earlier and some recent ideas about the possible modes of specification of the wiring-in of nervous systems are reviewed in the light of older and several recent experiments, and some new ideas are suggested. It is argued that certain general principles, notably the postulated ‘principle of alternative matching’ (PALMA) and a suggested and related ‘kaleidoscopic effect’ (KALEF), as well as the notion of an ‘extracellular guidance network’ (ECGN), are in good agreement with recent and older findings concerning axonal guidance during neural wiring-in. It seems possible that by means of genetically programmed processes, neurons become systematically combinatorially labelled to such a degree that possibly all neurons areuniquely specified, as regards the combination oftypes of cell labels they make. Yet, there remains considerable freedom as regards the modes of arrangements of cell labels within cell surface membranes and the KALEF permits to overcome apparent difficulties that confronted earlier versions of the cell labelling hypotheses (cf. Edelman,Science 219, 450–457, 1983, for mention of such difficulties). Apart from label specification, neural development seems to depend on trophic factors, which are also essential for the maintenance of the developed nervous system. The systematic programmes for cell labelling, apart from generating all the required neurons, also produces inappropriate neurons and synaptic connections. These are got rid of by systematic cell death and/or atrophy of inappropriate synapses and/or elimination of inappropriate axon collaterals. The resulting neural net seems then very specifically wired-in for each species, apparently without redundant neurons.     

Mathematical biophysics of some neural nets

Methods are considered for constructing neural nets of the McCulloch-Pitts type giving arbitrary time delays between stimuli and their responses. By introducing cycles of neurons, the problem is treated from the standpoint of economizing on the number of neurons required. A generalization is effect that will realize any unique temporal response pattern from a unique stimulus.     

A new model for simple neural nets and its application in the design of a neural oscillator

The results of a previous theoretical study of a class of systems are applied for the design of neural nets which try to simulate biological behavior. Besides the models for single aperiodic and periodic neurons, a “neural oscillator” is developed which consists of two cross-excited neurons. Its response is similar to the firing pattern of certain biological neural oscillators, like the flying system of the locust. Also, by proper change of its parameters, it can be made highly irregular, providing a deterministic model for the spontaneous neural activity.     

Evoked potentials in artificial neural nets

A neural net model of discrete populations of formal neurons have been constructed to study evoked potentials based on our previous simulation studies (Anninos, 1972). Some interesting results came up from the examination of our findings regarding the latencies and the period of the cyclic activity of the evoked potentials. In fact, the different successive latencies for the five identical stimuli and the different periods for each of the cyclic activities, all are consequences of inhibitory and excitatory influences from a large neuronal population. Furthermore, such behavior of the system is not only related to the unknown neuronal population but was also substantially altered by what occurred in other systems at the time of stimulus, or prior to it.Computation assistance was provided by the Health Sciences Computing Facility, UCLA, sponsored by NIH Special Research Resources Grant RR-3. Research was sponsored by NSF Grant GB 30498 and NIH Grant NS-8498.     

Noisy neural nets exhibiting memory domains

Previous studies with probabilistic neural nets in which the neural connections are set up by means of chemical markers, revealed the existence of multiple memory domains. We generalized these studies by considering the intrinsic noise of the systems, caused by the spontaneous release of synaptic transmitter substance. A simple mathematical model is developed, which yields characteristics of multiple memory domains analogous to those occurring in noiseless nets.     

Cyclic modes in artificial neural nets

Artificial neural nets constructed of dicrete populations of 200–1000 formal neurons have been studied through computer simulation. Among the basic assumptions of operation of these nets are the following: a) Each neuron fires at times which are integral multiples of the synaptic delay . b) It produces the appropriate PSP's after . c) All the neurons have the same refractory period and d) temporal summation occurs without decrement, for a period less than the synaptic delay. The nets were specified by a number of parameters: fraction of inhibitory neurons in the population, average number of connections to each cell, threshold for cell firing. These parameters did not determine the detailed microscopical structures of nets which was established separately on a random basis.For the range of the parameters considered in this study it was found that neural nets are capable of supporting self-maintaining activity in the form of cycling modes, characterized by a fixed period. The period of the cycles can be altered by a steady, non-cycling external input to the net. Evidence is presented that the cycling modes depend upon the statistical parameters of the net and the stimulus characteristics rather than on the detailed structure of the net. These results suggest that non-structured nerve nets may respond in specific manner to specific stimuli.

Glossary

Parameters of Neural Net Model Synaptic delay - A Total number of neurons in the netlet - h Fraction of inhibitory neurons in the netlet (in % of total number of neurons) - ⁺ Average number of axon branches emanating from anexcitatory neuron - ^– Average number of axon branches emanating from an inhibitory neuron - k ⁺ Average EPSP produced by an excitatory neuron in arbitrary units of amplitude - k ^– Average IPSP produced by an inhibitory neuron in arbitrary units of amplitude - Firing threshold of neurons in the netlet - The minimum number of ESPS's necessary to trigger a neuron in the absence of inhibitory inputs - The minimum number of ESPS's necessary to trigger a neuron in the presence of inhibitory inputs. Dynamic Parameters of the Model n An integer giving the number of elapsed synaptic delays (i.e. elapsed time) - _n The activity; i.e. the fraction of active neurons in the netlet at t=n (the actual number of active cells is given by _nA) - _n={ⁱ_n} State vector of single netlet at time n This research has been supported by NIH grants NS-8012 and NS-8498, and NSF grant GB-30498. Computation assistance was provoded by the Health Sciences Computing Facility, UCLA, sponsored by NIH Special Research Resources grant RR-3.     

Applications of matrix algebra to communication nets

A “generic” problem amenable to matrix algebraic treatment is outlined. Several examples are given and one, a communication system, is studied in some detail. A typical structure matrix is used to describe the channels of communication and a “status” matrix is used to describe the distribution of information in the system at any time. A theorem is proved relating the status matrix at any timet to thetth power of the structure matrix. The elements of the communication system are interpreted as individuals who can send messages to each other. For the individuals attempting to solve a “group problem” certain relations are derived between the structure and status matrices and time of solution. The structure of the communication system is permitted to vary with time. A general theorem is proved relating the status matrix to the matrix product of the series of structure matrices representing the changing structure of the system. Some suggestions are made for further generalizations. In particular, it is suggested that so-called “higher order” information transmission can be similarly treated.