Human learning characteristics in the tracking tasks of iterative nature

In this paper, human learning characteristics in the tracking tasks of iterative nature are investigated. Various linear and nonlinear systems are used as plant, and a human operator has to generate the proper control inputs to force these systems in tracking the desired trajectory. The learning behaviour of the human operator in modifying his control actions is studied and it is observed that the human operator can improve his performance quite efficiently despite the unavailability of any information about the system or the desired trajectories. It is concluded from the experiments that the human operator not only use the information that is directly available to him (error in this case), but also extracts some useful information (e.g. error rate) that he feels is necessary to generate a good control action. The limitation of the human performance is studied in frequency domain, and the performance of the human operator against the frequency bandwidth of error and error rate signals are highlighted. Analysis of the results revealed that a human operator gives more importance to the error rate in generating his control actions and, accordingly, it is observed that his limitation in term of performance is more sensitive to the frequency bandwidth of the error rate as compared to the error. The human operator cannot improve his performance once the frequency components of the error or error rates shift to the higher frequencies, say above 1.0 Hz.     

Coordinates transformation and learning control for visually-guided voluntary movement with iteration: A Newton-like method in a function space

In order to control visually-guided voluntary movements, the central nervous system (CNS) must solve the following three computational problems at different levels: (1) determination of a desired trajectory in the visual coordinates, (2) transformation of the coordinates of the desired trajectory to the body coordinates and (3) generation of motor command. In this paper, the second and the third problems are treated at computational, representational and hardware levels of Marr. We first study the problems at the computational level, and then propose an iterative learning scheme as a possible algorithm. This is a trial and error type learning such as repetitive training of golf swing. The amount of motor command needed to coordinate activities of many muscles is not determined at once, but in a step-wise, trial and error fashion in the course of a set of repetitions. Actually, the motor command in the (n+1)-th iteration is a sum of the motor command in then-th iteration plus two modification terms which are, respectively, proportional to acceleration and speed errors between the desired trajectory and the realized trajectory in then-th iteration. We mathematically formulate this iterative learning control as a Newton-like method in functional spaces and prove its convergence under appropriate mathematical conditions with use of dynamical system theory and functional analysis. Computer simulations of this iterative learning control of a robotic manipulator in the body or visual coordinates are shown. Finally, we propose that areas 2, 5, and 7 of the sensory association cortex are possible sites of this learning control. Further we propose neural network model which acquires transformation matrices from acceleration or velocity to motor command, which are used in these schemes.     

A mathematical model of the adaptive control of human arm motions

This paper discusses similarities between models of adaptive motor control suggested by recent experiments with human and animal subjects, and the structure of a new control law derived mathematically from nonlinear stability theory. In both models, the control actions required to track a specified trajectory are adaptively assembled from a large collection of simple computational elements. By adaptively recombining these elements, the controllers develop complex internal models which are used to compensate for the effects of externally imposed forces or changes in the physical properties of the system. On a motor learning task involving planar, multi-joint arm motions, the simulated performance of the mathematical model is shown to be qualitatively similar to observed human performance, suggesting that the model captures some of the interesting features of the dynamics of low-level motor adaptation. Received: 20 September 1994 / Accepted in revised form: 18 November 1998     

Spectral properties of human cognition and skill

 Many interactive human skills are based on real-time error detection and correction. Here we investigate the spectral properties of such skills, focusing on a synchronization task. A simple autoregressive error correction model, based on separate ‘motor’ and ‘cognitive’ sources, provides an excellent fit to experimental spectral data. The model can also apply to recurrent processes not based on error correction, allowing commentary on previous claims of 1/ f-type noise in human cognition. A comparison of expert and non-expert subjects suggests that performance skill is not only based on reduced variance and bias, but also on the construction of richer mental models of error correction. Received: 4 October 1995 / Accepted in revised form: 25 February 1997     

Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex

Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior.     

Statistical characteristics of climbing fiber spikes necessary for efficient cerebellar learning

 Mean firing rates (MFRs), with analogue values, have thus far been used as information carriers of neurons in most brain theories of learning. However, the neurons transmit the signal by spikes, which are discrete events. The climbing fibers (CFs), which are known to be essential for cerebellar motor learning, fire at the ultra-low firing rates (around 1 Hz), and it is not yet understood theoretically how high-frequency information can be conveyed and how learning of smooth and fast movements can be achieved. Here we address whether cerebellar learning can be achieved by CF spikes instead of conventional MFR in an eye movement task, such as the ocular following response (OFR), and an arm movement task. There are two major afferents into cerebellar Purkinje cells: parallel fiber (PF) and CF, and the synaptic weights between PFs and Purkinje cells have been shown to be modulated by the stimulation of both types of fiber. The modulation of the synaptic weights is regulated by the cerebellar synaptic plasticity. In this study we simulated cerebellar learning using CF signals as spikes instead of conventional MFR. To generate the spikes we used the following four spike generation models: (1) a Poisson model in which the spike interval probability follows a Poisson distribution, (2) a gamma model in which the spike interval probability follows the gamma distribution, (3) a max model in which a spike is generated when a synaptic input reaches maximum, and (4) a threshold model in which a spike is generated when the input crosses a certain small threshold. We found that, in an OFR task with a constant visual velocity, learning was successful with stochastic models, such as Poisson and gamma models, but not in the deterministic models, such as max and threshold models. In an OFR with a stepwise velocity change and an arm movement task, learning could be achieved only in the Poisson model. In addition, for efficient cerebellar learning, the distribution of CF spike-occurrence time after stimulus onset must capture at least the first, second and third moments of the temporal distribution of error signals. Received: 28 January 2000 / Accepted in revised form: 2 August 2000     

Prediction of MHC class II binding peptides based on an iterative learning model

Background

Prediction of the binding ability of antigen peptides to major histocompatibility complex (MHC) class II molecules is important in vaccine development. The variable length of each binding peptide complicates this prediction. Motivated by a text mining model designed for building a classifier from labeled and unlabeled examples, we have developed an iterative supervised learning model for the prediction of MHC class II binding peptides.

Results

A linear programming (LP) model was employed for the learning task at each iteration, since it is fast and can re-optimize the previous classifier when the training sets are altered. The performance of the new model has been evaluated with benchmark datasets. The outcome demonstrates that the model achieves an accuracy of prediction that is competitive compared to the advanced predictors (the Gibbs sampler and TEPITOPE). The average areas under the ROC curve obtained from one variant of our model are 0.753 and 0.715 for the original and homology reduced benchmark sets, respectively. The corresponding values are respectively 0.744 and 0.673 for the Gibbs sampler and 0.702 and 0.667 for TEPITOPE.

Conclusion

The iterative learning procedure appears to be effective in prediction of MHC class II binders. It offers an alternative approach to this important predictionproblem.     

Conclusions from a dynamic population model for tsetse: response to comments

The critique by Hargrove et al. (Popul Ecol, 2011) of our recently published paper on a tsetse population model (Barclay and Vreysen in Popul Ecol 53:89–110, 2011) has made some good points but has also misinterpreted the intent of some of our results as we presented them. Hargrove et al. rightly say that there is a mismatch between the size of the unit cells in the model (1 ha) and the iteration rate of the model (every 5 days), yielding too low a dispersal rate to simulate reality. However, they have misconstrued several of our results that we presented as examples to imply that those results were a necessary condition for control of tsetse, especially using traps and targets.     

Impedance characteristics of a neuromusculoskeletal model of the human arm I. Posture control

The mechanical impedance of neuromusculoskeletal models of the human arm is studied in this paper. The model analysis provides a better understanding of the contributions of possible intrinsic and reflexive components of arm impedance, makes clear the limitations of second-order mass-viscosity-stiffness models and reveals possible task effects on the impedance. The musculoskeletal model describes planar movements of the upper arm and forearm, which are moved by six lumped muscles with nonlinear dynamics. The motor control system is represented by a neural network which combines feedforward and feedback control. It is optimized for the control of movements or for posture control in the presence of external forces. The achieved impedance characteristics depend on the conditions during the learning process. In particular, the impedance is adapted in a suitable way to the frequency content and direction of external forces acting on the hand during an isometric task. The impedance characteristics of a model, which is optimized for movement control, are similar to experimental data in the literature. The achieved stiffness is, to a large extent, reflexively determined whereas the approximated viscosity is primarily due to intrinsic attributes. It is argued that usually applied Hill-type muscle models do not properly represent intrinsic muscle stiffness. Received: 14 October 1997 / Accepted in revised form: 18 May 1999     

Simulation of an iterative learning control system for fed-batch cell culture processes

This paper describes an iterative learning control scheme for fed-batch operation where repetitive trajectory tracking tasks are required. The proposed learning strategy is model-independent, and it takes advantage of the repetitive feature of system operations with a certain degree of intelligence and requires only small size of dynamic database for the learning process. The convergence of the learning process is proven. An example of simultaneously tracking two predefined trajectories by iterative learning control with two control inputs is given to illustrate the methodology. Satisfactory performance of the learning system can be observed from the simulation results.     

Characteristics of Human Luminance Discrimination and Modeling a Neural Network Based on the Response Properties of the Visual Cortex

Reaction time (RT) and error rate that depend on stimulus duration were measured in a luminance-discrimination reaction time task. Two patches of light with different luminance were presented to participants for ‘short’ (150 ms) or ‘long’ (1 s) period on each trial. When the stimulus duration was ‘short’, the participants responded more rapidly with poorer discrimination performance than they did in the longer duration. The results suggested that different sensory responses in the visual cortices were responsible for the dependence of response speed and accuracy on the stimulus duration during the luminance-discrimination reaction time task. It was shown that the simple winner-take-all-type neural network model receiving transient and sustained stimulus information from the primary visual cortex successfully reproduced RT distributions for correct responses and error rates. Moreover, temporal spike sequences obtained from the model network closely resembled to the neural activity in the monkey prefrontal or parietal area during other visual decision tasks such as motion discrimination and oddball detection tasks.     

A model of activity-dependent formation of cerebellar microzones

According to modern views of the cerebellum in motor control, each cerebellar functional unit, or microzone, learns how to execute predictive and coordinative control, based on long-term depression of the granule cell-Purkinje cell synapses. In the present paper, in light of recent experimental and theoretical studies on synaptic elimination and cerebellar motor learning, a model of the formation of cerebellar microzones by climbing fiber synaptic elimination is proposed. It is shown that competition for an activity-dependent supply of neurotrophic factor can reproduce the spatio-temporal characteristics of climbing fiber synaptic elimination. It is further shown that when this elimination is accurate, motor coordination can be acquired in an arm reaching task. In view of the results of the present study, several predictions are proposed. Received: 19 January 1998 / Accepted in revised form: 22 April 1998     

A distributed control model for the air-righting reflex of a cat

A multisegment, multijoint model of a falling animal is presented to examine the effectiveness of a two-stage control scheme in a zero-momentum self-righting maneuver. The model contains a much larger number of degrees of freedom than is required to execute a self-righting maneuver and is thus capable of providing multiple solutions for the same task. The decentralized control scheme is designed to achieve gross turning in minimum time and to maintain a steady orientation relative to gravity after the turn has been achieved. The scheme is able to determine the sequence of steps necessary to execute the motor task and also incorporates learning features. Results from various simulations are presented and their implications discussed. Received: 26 June 1995 / Accepted in revised form: 30 June 1998     

Modeling perceptual learning: difficulties and how they can be overcome

We investigated the roles of feedback and attention in training a vernier discrimination task as an example of perceptual learning. Human learning even of simple stimuli, such as verniers, relies on more complex mechanisms than previously expected – ruling out simple neural network models. These findings are not just an empirical oddity but are evidence that present models fail to reflect some important characteristics of the learning process. We will list some of the problems of neural networks and develop a new model that solves them by incorporating top-down mechanisms. Contrary to neural networks, in our model learning is not driven by the set of stimuli only. Internal estimations of performance and knowledge about the task are also incorporated. Our model implies that under certain conditions the detectability of only some of the stimuli is enhanced while the overall improvement of performance is attributed to a change of decision criteria. An experiment confirms this prediction. Received: 23 May 1996 / Accepted in revised form: 16 October 1997     

Impaired associative learning in schizophrenia: behavioral and computational studies

Associative learning is a central building block of human cognition and in large part depends on mechanisms of synaptic plasticity, memory capacity and fronto–hippocampal interactions. A disorder like schizophrenia is thought to be characterized by altered plasticity, and impaired frontal and hippocampal function. Understanding the expression of this dysfunction through appropriate experimental studies, and understanding the processes that may give rise to impaired behavior through biologically plausible computational models will help clarify the nature of these deficits. We present a preliminary computational model designed to capture learning dynamics in healthy control and schizophrenia subjects. Experimental data was collected on a spatial-object paired-associate learning task. The task evinces classic patterns of negatively accelerated learning in both healthy control subjects and patients, with patients demonstrating lower rates of learning than controls. Our rudimentary computational model of the task was based on biologically plausible assumptions, including the separation of dorsal/spatial and ventral/object visual streams, implementation of rules of learning, the explicit parameterization of learning rates (a plausible surrogate for synaptic plasticity), and learning capacity (a plausible surrogate for memory capacity). Reductions in learning dynamics in schizophrenia were well-modeled by reductions in learning rate and learning capacity. The synergy between experimental research and a detailed computational model of performance provides a framework within which to infer plausible biological bases of impaired learning dynamics in schizophrenia.     

Examination of the λ equilibrium point hypothesis when applied to single degree of freedom movements performed with different inertial loads

 One of the theories of human motor control is the λ Equilibrium Point Hypothesis. It is an attractive theory since it offers an easy control scheme where the planned trajectory shifts monotionically from an initial to a final equilibrium state. The feasibility of this model was tested by reconstructing the virtual trajectory and the stiffness profiles for movements performed with different inertial loads and examining them. Three types of movements were tested: passive movements, targeted movements, and repetitive movements. Each of the movements was performed with five different inertial loads. Plausible virtual trajectories and stiffness profiles were reconstructed based on the λ Equilibrium Point Hypothesis for the three different types of movements performed with different inertial loads. However, the simple control strategy supported by the model, where the planned trajectory shifts monotonically from an initial to a final equilibrium state, could not be supported for targeted movements performed with added inertial load. To test the feasibility of the model further we must examine the probability that the human motor control system would choose a trajectory more complicated than the actual trajectory to control. Received: 20 June 1995 / Accepted in revised form: 6 August 1996     

Stimulus enhancement in greylag geese: socially mediated learning of an operant task

We recently observed the spreading of a novel tradition in a flock of semiferal greylag geese, Anser anser: an increasing number of individuals began to bite and chew the stems of butterbur, Petasites hybridus. Because this behaviour spread particularly fast within families, social learning seemed to be involved. We therefore designed an experiment with hand-reared goslings, which were socially imprinted on humans, to investigate whether and how the observation of an experienced tutor affects the acquisition of a novel skill. Goslings had to open the gliding lid of a box to get at a food reward. To each of seven hand-reared observers a human tutor demonstrated where and how to open the lid, whereas seven controls remained untutored. All observers learned to perform the task but only one of the controls succeeded. The observers explored more often at the position shown by the tutor than elsewhere and seemingly learned by trial and error. In contrast, control birds explored primarily at positions that did not allow them to open the box. These results indicate that in greylag goslings the observation of an experienced model facilitates the learning of an operant task. We conclude that stimulus enhancement followed by operant conditioning were the mechanisms involved, which may have accounted for the fast spread of the stem-chewing tradition between family members. Copyright 2000 The Association for the Study of Animal Behaviour.     

An approach for developing an experimentally based model for simulating flight-phase dynamics

 This paper presents an approach for developing an experimentally validated dynamic multisegment model to simulate human flight-phase dynamics and multijoint control. Modeling and experimental techniques were integrated to systematically examine the contribution of multiple error sources to the accuracy of the model and to determine the complexity of a model that adequately emulates the dynamic behavior at the total-body and multijoint levels during flight. The accuracy of the model and of the experimental data was assessed using an inverse dynamics simulation of flight-phase motion for two representative cases: (i) a physical model released from a bar and (ii) a gymnast performing a layout dismount from a bar. Multijoint models with varying numbers of segments were assessed in order to determine the complexity of the model that adequately simulates the flight-phase task. A five-segment model was found to adequately simulate the layout dismount performed by the gymnast. The error introduced during modeling and digitizing contributed to an apparent violation of the conservation law manifested as large external forces acting on the nonactuated joints. These results demonstrate the need to reduce sources of error prior to testing hypotheses regarding feedforward and feedback components of the multijoint control system. The proposed approach for quantifying sources of error provides a crucial step that is required in the development of experimentally based dynamic models designed to examine and test hypotheses regarding multijoint control logic. Received: 5 April 2001 / Accepted in revised form: 25 April 2002 Acknowledgements. This work was funded by Intel, a dissertation awarded from the International Society of Biomechanics, the Internationale Federation de Gymnastique, the International Olympic Committee, and Pfizer. We express our special thanks to Kathleen Costa and Witaya Mathiyakom for their assistance in data collection. Correspondence to: P. S. Requejo (e-mail: requejo@rcf.usc.edu)     

Adaptive dynamic programming as a theory of sensorimotor control

Many characteristics of sensorimotor control can be explained by models based on optimization and optimal control theories. However, most of the previous models assume that the central nervous system has access to the precise knowledge of the sensorimotor system and its interacting environment. This viewpoint is difficult to be justified theoretically and has not been convincingly validated by experiments. To address this problem, this paper presents a new computational mechanism for sensorimotor control from a perspective of adaptive dynamic programming (ADP), which shares some features of reinforcement learning. The ADP-based model for sensorimotor control suggests that a command signal for the human movement is derived directly from the real-time sensory data, without the need to identify the system dynamics. An iterative learning scheme based on the proposed ADP theory is developed, along with rigorous convergence analysis. Interestingly, the computational model as advocated here is able to reproduce the motor learning behavior observed in experiments where a divergent force field or velocity-dependent force field was present. In addition, this modeling strategy provides a clear way to perform stability analysis of the overall system. Hence, we conjecture that human sensorimotor systems use an ADP-type mechanism to control movements and to achieve successful adaptation to uncertainties present in the environment.