A neural model for conditioned avoidance learning

A mathematical model for learning of a conditioned avoidance behavior is presented. An identification of the net excitation of a neural model (Rashevsky, N., 1960.Mathematical Biophysics. Vol. II. New York: Dover Publications, Inc.) with the instantaneous probability of response is introduced and its usefulness in discussing block-trial learning performances in the conditioned avoidance situation is outlined for normal and brain-operated animals, using experimental data collected by the author. Later, the model is applied to consecutive trial learning and connection is made with the approach of H. D. Landahl (1964. “An Avoidance Learning Situation. A Neural Net Model.”Bull. Math. Biophysics,26, 83–89; and 1965, “A Neural Net Model for Escape Learning.”Bull. Math. Biophysics,27, Special Edition, 317–328) wherein lie further data with which the model can be compared.     

A neural net model for escape learning

An escape learning situation is discussed in terms of a neural model in which a stimulus can result in a conditioned excitement and a specific conditioned response. By using the simplest relations between the strengths of conditioning and the number of reinforcements and by introducing a distribution of fluctuations occurring regularly in time, one can calculate the probabilities of various responses, as well as the various latencies, in successive trials. The results are in moderately satisfactory agreement with the data of R. L. Solomon and L. C. Wynne (Psychol. Monogr.,67, No. 4, 1953). Consequences of the model for various experimental situations are discussed. This research was supported in part by the United States Public Health Service Grant RCA GM K6 18,420 and in part by the United States Air Force through the Air Force Office of Scientific Research of the Air Research Development Command under Grant No. AF AFOSR 370-64.     

A neural model for generating and learning a rapid movement sequence

 In this article, a neural model for generating and learning a rapid ballistic movement sequence in two-dimensional (2D) space is presented and evaluated in the light of some considerations about handwriting generation. The model is based on a central nucleus (called a planning space) consisting of a fully connected grid of leaky integrators simulating neurons, and reading an input vector Ξ (t) which represents the external movement of the end effector. The movement sequencing results in a succession of motor strokes whose instantiation is controlled by the global activation of the planning space as defined by a competitive interaction between the neurons of the grid. Constraints such as spatial accuracy and movement time are exploited for the correct synchronization of the impulse commands. These commands are then fed into a neuromuscular synergy whose output is governed by a delta lognormal equation. Each movement sequence is memorized originally as a symbolic engram representing the sequence of the principal reference points of the 2D movement. These points, called virtual targets, correspond to the targets of each single rapid motor stroke composing the movement sequence. The task during the learning phase is to detect the engram corresponding to a new observed movement; the process is controlled by the dynamics of the neural grid. Received: 16 March 1995/Accepted in revised form: 25 July 1995     

A model for complex sequence learning and reproduction in neural populations

Temporal patterns of activity which repeat above chance level in the brains of vertebrates and in the mammalian neocortex have been reported experimentally. This temporal structure is thought to subserve functions such as movement, speech, and generation of rhythms. Several studies aim to explain how particular sequences of activity are learned, stored, and reproduced. The learning of sequences is usually conceived as the creation of an excitation pathway within a homogeneous neuronal population, but models embodying the autonomous function of such a learning mechanism are fraught with concerns about stability, robustness, and biological plausibility. We present two related computational models capable of learning and reproducing sequences which come from external stimuli. Both models assume that there exist populations of densely interconnected excitatory neurons, and that plasticity can occur at the population level. The first model uses temporally asymmetric Hebbian plasticity to create excitation pathways between populations in response to activation from an external source. The transition of the activity from one population to the next is permitted by the interplay of excitatory and inhibitory populations, which results in oscillatory behavior that seems to agree with experimental findings in the mammalian neocortex. The second model contains two layers, each one like the network used in the first model, with unidirectional excitatory connections from the first to the second layer experiencing Hebbian plasticity. Input sequences presented in the second layer become associated with the ongoing first layer activity, so that this activity can later elicit the the presented sequence in the absence of input. We explore the dynamics of these models, and discuss their potential implications, particularly to working memory, oscillations, and rhythm generation.     

An avoidance learning situation. A neural net model

A simple avoidance situation is considered in terms of a neural net learning model. Data for the control situation can be represented by an expression having three parameters which determine the initial and the steady state activities together with the transient aspects. The introduction of a learning parameter then allows one to calculate satisfactorily the results obtained in the experimental situation in which shock is applied. This research was supported in part by the United States Air Force through the Air Force Office of Scientific Research of the Air Research Development Command under Grant No. AF AFOSR 370-63 and in part by the United States Public Health Service Grant RCA GM K6 18,420.     

A learning network model of the neural integrator of the oculomotor system

Certain premotor neurons of the oculomotor system fire at a rate proportional to desired eye velocity. Their output is integrated by a network of neurons to supply an eye positon command to the motoneurons of the extraocular muscles. This network, known as the neural integrator, is calibrated during infancy and then maintained through development and trauma with remarkable precision. We have modeled this system with a self-organizing neural network that learns to integrate vestibular velocity commands to generate appropriate eye movements. It learns by using current eye movement on any given trial to calculate the amount of retinal image slip and this is used as the error signal. The synaptic weights are then changed using a straightforward algorithm that is independent of the network configuration and does not necessitate backwards propagation of information. Minimization of the error in this fashion causes the network to develop multiple positive feedback loops that enable it to integrate a push-pull signal without integrating the background rate on which it rides. The network is also capable of recovering from various lesions and of generating more complicated signals to simulate induced postsaccadic drift and compensation for eye muscle mechanics.     

A neural net model for epilepsy

A neural net model based in our previous studies with randomly interconnected neural nets is presented here capable of exhibiting epileptic features. These features can be explained in terms of the structural and dynamical properties of the model. In addition, apart from the fact that this model can imitate epileptic phenomena, it might also help to explain some poorly understood clinical phenomena from which general disturbances can produce focal seizures in the brain.     

Disjunctive models of Boolean category learning

Four connectionistic/neural models which are capable of learning arbitrary Boolean functions are presented. Three are provably convergent, but of differing generalization power. The fourth is not necessarily convergent, but its empirical behavior is quite good. The time and space characteristics of the four models are compared over a diverse range of functions and testing conditions. These include the ability to learn specific instances, to effectively generalize, and to deal with irrelevant or redundant information. Trade-offs between time and space are demonstrated by the various approaches.     

Modeling perceptual learning with multiple interacting elements: A neural network model describing early visual perceptual learning

We introduce a neural network model of an early visual cortical area, in order to understand better results of psychophysical experiments concerning perceptual learning during odd element (pop-out) detection tasks (Ahissar and Hochstein, 1993, 1994a).The model describes a network, composed of orientation selective units, arranged in a hypercolumn structure, with receptive field properties modeled from real monkey neurons. Odd element detection is a final pattern of activity with one (or a few) salient units active. The learning algorithm used was the Associative reward-penalty (Ar-p) algorithm of reinforcement learning (Barto and Anandan, 1985), following physiological data indicating the role of supervision in cortical plasticity.Simulations show that network performance improves dramatically as the weights of inter-unit connections reach a balance between lateral iso-orientation inhibition, and facilitation from neighboring neurons with different preferred orientations. The network is able to learn even from chance performance, and in the presence of a large amount of noise in the response function. As additional tests of the model, we conducted experiments with human subjects in order to examine learning strategy and test model predictions.     

A neural net model for masking phenomena

Some aspects of masking phenomena are considered in terms of the simplest possible model of two-factor neural elements. The effect of a number of variables can be accounted for, but the introduction of an internuncial element results in a masking function which need not be symmetric about zero delay interval. As an illustration, the results for a special case are compared with available data. In general, such a model results in a masking function which depends on the intensity, area, and duration of the stimuli, as well as on the temporal and spatial separation between them.     

A neural network model for cognitive activity

A consideration of the storage of information as an energized neuronal state leads to the development of a new type of neural network model which is capable of pattern recognition, concept formation and recognition of patterns of events in time. The network consists of several layers of cells, each cell representing by connections from the lower levels some combination of features or concepts. Information travels toward higher layers by such connections during an association phase, and then reverses during a recognition phase, where higher-order concepts can redirect the flow to more appropriate elements, revising the perception of the environment. This permits a more efficient method of distinguishing closely-related patterns and also permits the formation of negative associations, which is a likely requirement for formation of "abstract" concepts.     

A neural network model for texture discrimination

A model of texture discrimination in visual cortex was built using a feedforward network with lateral interactions among relatively realistic spiking neural elements. The elements have various membrane currents, equilibrium potentials and time constants, with action potentials and synapses. The model is derived from the modified programs of MacGregor (1987). Gabor-like filters are applied to overlapping regions in the original image; the neural network with lateral excitatory and inhibitory interactions then compares and adjusts the Gabor amplitudes in order to produce the actual texture discrimination. Finally, a combination layer selects and groups various representations in the output of the network to form the final transformed image material. We show that both texture segmentation and detection of texture boundaries can be represented in the firing activity of such a network for a wide variety of synthetic to natural images. Performance details depend most strongly on the global balance of strengths of the excitatory and inhibitory lateral interconnections. The spatial distribution of lateral connective strengths has relatively little effect. Detailed temporal firing activities of single elements in the lateral connected network were examined under various stimulus conditions. Results show (as in area 17 of cortex) that a single element's response to image features local to its receptive field can be altered by changes in the global context.     

A continuous-time neural model for sequential action

Action selection, planning and execution are continuous processes that evolve over time, responding to perceptual feedback as well as evolving top-down constraints. Existing models of routine sequential action (e.g. coffee- or pancake-making) generally fall into one of two classes: hierarchical models that include hand-built task representations, or heterarchical models that must learn to represent hierarchy via temporal context, but thus far lack goal-orientedness. We present a biologically motivated model of the latter class that, because it is situated in the Leabra neural architecture, affords an opportunity to include both unsupervised and goal-directed learning mechanisms. Moreover, we embed this neurocomputational model in the theoretical framework of the theory of event coding (TEC), which posits that actions and perceptions share a common representation with bidirectional associations between the two. Thus, in this view, not only does perception select actions (along with task context), but actions are also used to generate perceptions (i.e. intended effects). We propose a neural model that implements TEC to carry out sequential action control in hierarchically structured tasks such as coffee-making. Unlike traditional feedforward discrete-time neural network models, which use static percepts to generate static outputs, our biological model accepts continuous-time inputs and likewise generates non-stationary outputs, making short-timescale dynamic predictions.     

A neural network model for trace conditioning

We studied the dynamics of a neural network that has both recurrent excitatory and random inhibitory connections. Neurons started to become active when a relatively weak transient excitatory signal was presented and the activity was sustained due to the recurrent excitatory connections. The sustained activity stopped when a strong transient signal was presented or when neurons were disinhibited. The random inhibitory connections modulated the activity patterns of neurons so that the patterns evolved without recurrence with time. Hence, a time passage between the onsets of the two transient signals was represented by the sequence of activity patterns. We then applied this model to represent the trace eye blink conditioning, which is mediated by the hippocampus. We assumed this model as CA3 of the hippocampus and considered an output neuron corresponding to a neuron in CA1. The activity pattern of the output neuron was similar to that of CA1 neurons during trace eye blink conditioning, which was experimentally observed.     

A neural model for nonassociative learning in a prototypical sensory-motor scheme: the landing reaction in flies

Nonassociative learning is an important property of neural organization in both vertebrate and invertebrate species. In this paper we propose a neural model for nonassociative learning in a well studied prototypical sensory-motor scheme: the landing reaction of flies. The general structure of the model consists of sensory processing stages, a sensory-motor gate network, and motor control circuits. The paper concentrates on the sensory-motor gate network which has an agonist-antagonist structure. Sensory inputs to this circuit are transduced by chemical messenger systems whose dynamics include depletion and replenishment terms. The resulting circuit is a gated dipole anatomy and we show that it gives a good account of nonassociative learning in the landing reaction of the fly.Supported by a grant from the National Institute of Mental Health     

A real time learning algorithm for recurrent analog neural networks

A new learning algorithm is described for a general class of recurrent analog neural networks which ultimately settle down to a steady state. Recently, Pineda (Pineda 1987; Almeida 1987; Ikeda et al. 1988) has introduced a learning rule for the recurrent net in which the connection weights are adjusted so that the distance between the stable outputs of the current system and the desired outputs will be maximally decreased. In this method, many cycles are needed in order to get a target system. In each cycle, the recurrent net is run until it reaches a stable state. After that, the weight change is calculated by using a linearized recurrent net which receives the current error of the system as a bias input. In the new algorithm the weights are changed so that the total error of neuron outputs over the entire trajectory is minimized. The weights are adjusted in real time when the network is running. In this method, the trajectory to the target system can be controlled, whereas Pineda's algorithm only controls the position of the fixed point. The relation to the back propagation method (Hinton et al. 1986) is also discussed.     

A model for learning human reaching movements

 Reaching movement is a fast movement towards a given target. The main characteristics of such a movement are straight path and a bell-shaped speed profile. In this work a mathematical model for the control of the human arm during ballistic reaching movements is presented. The model of the arm contains a 2 degrees of freedom planar manipulator, and a Hill-type, non-linear mechanical model of six muscles. The arm model is taken from the literature with minor changes. The nervous system is modeled as an adjustable pattern generator that creates the control signals to the muscles. The control signals in this model are rectangular pulses activated at various amplitudes and timings, that are determined according to the given target. These amplitudes and timings are the parameters that should be related to each target and initial conditions in the workspace. The model of the nervous system consists of an artificial neural net that maps any given target to the parameter space of the pattern generator. In order to train this net, the nervous system model includes a sensitivity model that transforms the error from the arm end-point coordinates to the parameter coordinates. The error is assessed only at the termination of the movement from knowledge of the results. The role of the non-linearity in the muscle model and the performance of the learning scheme are analysed, illustrated in simulations and discussed. The results of the present study demonstrate the central nervous system’s (CNS) ability to generate typical reaching movements with a simple feedforward controller that controls only the timing and amplitude of rectangular excitation pulses to the muscles and adjusts these parameters based on knowledge of the results. In this scheme, which is based on the adjustment of only a few parameters instead of the whole trajectory, the dimension of the control problem is reduced significantly. It is shown that the non-linear properties of the muscles are essential to achieve this simple control. This conclusion agrees with the general concept that motor control is the result of an interaction between the nervous system and the musculoskeletal dynamics. Received : 21 May 1996 / Accepted in revised form : 10 June 1997     

A model for sensorimotor control and learning

A model for motor learning, generalization, and adaptation is presented. It is shown that the equations of motion of a limb can be expressed in a parametric form that facilitates transformation of desired trajectories into plans. These parametric equations are used in conjunction with a quantized multidimensional memory organized by state variables. The memory is supplied with data derived from the analysis of practice movements. A small computer and mechanical arm are used to implement the model and study its properties. Results verify the ability to acquire new movements, adapt to mechanical loads, and generalize between similar movements.This research was done while the author was a graduate student at the Massachusetts Institute of Technology in the Artificial Intelligence Laboratory and Department of Psychology. It was supported in part by training grant NGMS 5-T01-GM01064-15     

A neural signature of hierarchical reinforcement learning

Human behavior displays hierarchical structure: simple actions cohere into subtask sequences, which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive prediction arising from HRL. In ordinary reinforcement learning, reward prediction errors are computed when there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In three neuroimaging studies we observed neural responses consistent with such subgoal-related reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.