Interacting adaptive processes with different timescales underlie short-term motor learning

Multiple processes may contribute to motor skill acquisition, but it is thought that many of these processes require sleep or the passage of long periods of time ranging from several hours to many days or weeks. Here we demonstrate that within a timescale of minutes, two distinct fast-acting processes drive motor adaptation. One process responds weakly to error but retains information well, whereas the other responds strongly but has poor retention. This two-state learning system makes the surprising prediction of spontaneous recovery (or adaptation rebound) if error feedback is clamped at zero following an adaptation-extinction training episode. We used a novel paradigm to experimentally confirm this prediction in human motor learning of reaching, and we show that the interaction between the learning processes in this simple two-state system provides a unifying explanation for several different, apparently unrelated, phenomena in motor adaptation including savings, anterograde interference, spontaneous recovery, and rapid unlearning. Our results suggest that motor adaptation depends on at least two distinct neural systems that have different sensitivity to error and retain information at different rates.     

Reexposure to a sensorimotor perturbation produces opposite effects on explicit and implicit learning processes

The motor system demonstrates an exquisite ability to adapt to changes in the environment and to quickly reset when these changes prove transient. If similar environmental changes are encountered in the future, learning may be faster, a phenomenon known as savings. In studies of sensorimotor learning, a central component of savings is attributed to the explicit recall of the task structure and appropriate compensatory strategies. Whether implicit adaptation also contributes to savings remains subject to debate. We tackled this question by measuring, in parallel, explicit and implicit adaptive responses in a visuomotor rotation task, employing a protocol that typically elicits savings. While the initial rate of learning was faster in the second exposure to the perturbation, an analysis decomposing the 2 processes showed the benefit to be solely associated with explicit re-aiming. Surprisingly, we found a significant decrease after relearning in aftereffect magnitudes during no-feedback trials, a direct measure of implicit adaptation. In a second experiment, we isolated implicit adaptation using clamped visual feedback, a method known to eliminate the contribution of explicit learning processes. Consistent with the results of the first experiment, participants exhibited a marked reduction in the adaptation function, as well as an attenuated aftereffect when relearning from the clamped feedback. Motivated by these results, we reanalyzed data from prior studies and observed a consistent, yet unappreciated pattern of attenuation of implicit adaptation during relearning. These results indicate that explicit and implicit sensorimotor processes exhibit opposite effects upon relearning: Explicit learning shows savings, while implicit adaptation becomes attenuated

Humans learning a new motor task typically improve with repeated practice due to the faster expression of more effective explicit strategies; this study reveals that when motor learning occurs without awareness, performance deteriorates upon relearning.     

Estimating the relevance of world disturbances to explain savings, interference and long-term motor adaptation effects

Recent studies suggest that motor adaptation is the result of multiple, perhaps linear processes each with distinct time scales. While these models are consistent with some motor phenomena, they can neither explain the relatively fast re-adaptation after a long washout period, nor savings on a subsequent day. Here we examined if these effects can be explained if we assume that the CNS stores and retrieves movement parameters based on their possible relevance. We formalize this idea with a model that infers not only the sources of potential motor errors, but also their relevance to the current motor circumstances. In our model adaptation is the process of re-estimating parameters that represent the body and the world. The likelihood of a world parameter being relevant is then based on the mismatch between an observed movement and that predicted when not compensating for the estimated world disturbance. As such, adapting to large motor errors in a laboratory setting should alert subjects that disturbances are being imposed on them, even after motor performance has returned to baseline. Estimates of this external disturbance should be relevant both now and in future laboratory settings. Estimated properties of our bodies on the other hand should always be relevant. Our model demonstrates savings, interference, spontaneous rebound and differences between adaptation to sudden and gradual disturbances. We suggest that many issues concerning savings and interference can be understood when adaptation is conditioned on the relevance of parameters.     

The Decay of Motor Memories Is Independent of Context Change Detection

When the error signals that guide human motor learning are withheld following training, recently-learned motor memories systematically regress toward untrained performance. It has previously been hypothesized that this regression results from an intrinsic volatility in these memories, resulting in an inevitable decay in the absence of ongoing error signals. However, a recently-proposed alternative posits that even recently-acquired motor memories are intrinsically stable, decaying only if a change in context is detected. This new theory, the context-dependent decay hypothesis, makes two key predictions: (1) after error signals are withheld, decay onset should be systematically delayed until the context change is detected; and (2) manipulations that impair detection by masking context changes should result in prolonged delays in decay onset and reduced decay amplitude at any given time. Here we examine the decay of motor adaptation following the learning of novel environmental dynamics in order to carefully evaluate this hypothesis. To account for potential issues in previous work that supported the context-dependent decay hypothesis, we measured decay using a balanced and baseline-referenced experimental design that allowed for direct comparisons between analogous masked and unmasked context changes. Using both an unbiased variant of the previous decay onset analysis and a novel highly-powered group-level version of this analysis, we found no evidence for systematically delayed decay onset nor for the masked context change affecting decay amplitude or its onset time. We further show how previous estimates of decay onset latency can be substantially biased in the presence of noise, and even more so with correlated noise, explaining the discrepancy between the previous results and our findings. Our results suggest that the decay of motor memories is an intrinsic feature of error-based learning that does not depend on context change detection.     

Compensation for changing motor uncertainty

When movement outcome differs consistently from the intended movement, errors are used to correct subsequent movements (e.g., adaptation to displacing prisms or force fields) by updating an internal model of motor and/or sensory systems. Here, we examine changes to an internal model of the motor system under changes in the variance structure of movement errors lacking an overall bias. We introduced a horizontal visuomotor perturbation to change the statistical distribution of movement errors anisotropically, while monetary gains/losses were awarded based on movement outcomes. We derive predictions for simulated movement planners, each differing in its internal model of the motor system. We find that humans optimally respond to the overall change in error magnitude, but ignore the anisotropy of the error distribution. Through comparison with simulated movement planners, we found that aimpoints corresponded quantitatively to an ideal movement planner that updates a strictly isotropic (circular) internal model of the error distribution. Aimpoints were planned in a manner that ignored the direction-dependence of error magnitudes, despite the continuous availability of unambiguous information regarding the anisotropic distribution of actual motor errors.     

Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses

 Some characteristics of arm movements that humans exhibit during learning the dynamics of reaching are consistent with a theoretical framework where training results in motor commands that are gradually modified to predict and compensate for novel forces that may act on the hand. As a first approximation, the motor control system behaves as an adapting controller that learns an internal model of the dynamics of the task. It approximates inverse dynamics and predicts motor commands that are appropriate for a desired limb trajectory. However, we had previously noted that subtle motion characteristics observed during changes in task dynamics challenged this simple model and raised the possibility that adaptation also involved sensory–motor feedback pathways. These pathways reacted to sensory feedback during the course of the movement. Here we hypothesize that adaptation to dynamics might also involve a modification of how the CNS responds to sensory feedback. We tested this through experiments that quantified how the motor system's response to errors during voluntary movements changed as it adapted to dynamics of a force field. We describe a nonlinear approach that approximates the impedance of the arm, i.e., force response as a function of arm displacement trajectory. We observe that after adaptation, the impedance function changes in a way that closely matches and counters the effect of the force field. This is particularly prominent in the long-latency (>100 ms) component of response to perturbations. Therefore, it appears that practice not only modifies the internal model with which the brain generates motor commands that initiate a movement, but also the internal model with which sensory feedback is integrated with the ongoing descending commands in order to respond to error during the movement. Received: 10 January 2001 / Accepted in revised form: 30 May 2001     

The mind through chick eyes: memory,cognition and anticipation

To understand the animal mind, we have to reconstruct how animals recognize the external world through their own eyes. For the reconstruction to be realistic, explanations must be made both in their proximate causes (brain mechanisms) as well as ultimate causes (evolutionary backgrounds). Here, we review recent advances in the behavioral, psychological, and system-neuroscience studies accomplished using the domestic chick as subjects. Diverse behavioral paradigms are compared (such as filial imprinting, sexual imprinting, one-trial passive avoidance learning, and reinforcement operant conditioning) in their behavioral characterizations (development, sensory and motor aspects of functions, fitness gains) and relevant brain mechanisms. We will stress that common brain regions are shared by these distinct paradigms, particularly those in the ventral telencephalic structures such as AIv (in the archistriatum) and LPO (in the medial striatum). Neuronal ensembles in these regions could code the chick's anticipation for forthcoming events, particularly the quality/quantity and the temporal proximity of rewards. Without the internal representation of the anticipated proximity in LPO, behavioral tolerance will be lost, and the chick makes impulsive choice for a less optimized option. Functional roles of these regions proved compatible with their anatomical counterparts in the mammalian brain, thus suggesting that the neural systems linking between the memorized past and the anticipated future have remained highly conservative through the evolution of the amniotic vertebrates during the last 300 million years. With the conservative nature in mind, research efforts should be oriented toward a unifying theory, which could explain behavioral deviations from optimized foraging, such as "na?ve curiosity," "contra-freeloading," "Concorde fallacy," and "altruism."     

Hierarchical motor adaptations negotiate failures during force field learning

Humans have the amazing ability to learn the dynamics of the body and environment to develop motor skills. Traditional motor studies using arm reaching paradigms have viewed this ability as the process of ‘internal model adaptation’. However, the behaviors have not been fully explored in the case when reaches fail to attain the intended target. Here we examined human reaching under two force fields types; one that induces failures (i.e., target errors), and the other that does not. Our results show the presence of a distinct failure-driven adaptation process that enables quick task success after failures, and before completion of internal model adaptation, but that can result in persistent changes to the undisturbed trajectory. These behaviors can be explained by considering a hierarchical interaction between internal model adaptation and the failure-driven adaptation of reach direction. Our findings suggest that movement failure is negotiated using hierarchical motor adaptations by humans.     

Effects of subclinical depression on prefrontal–striatal model-based and model-free learning

Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal–striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.     

Cerebellar complex spike firing is suitable to induce as well as to stabilize motor learning

BACKGROUND: Cerebellar Purkinje cells (PC) generate two responses: the simple spike (SS), with high firing rates (>100 Hz), and the complex spike (CS), characterized by conspicuously low discharge rates (1-2 Hz). Contemporary theories of cerebellar learning suggest that the CS discharge pattern encodes an error signal that drives changes in SS activity, ultimately related to motor behavior. This then predicts that CS will discharge in relation to the error and at random once the error has been nulled by the new behavior. RESULTS: We tested this hypothesis with saccadic adaptation in macaque monkeys as a model of cerebellar-dependent motor learning. During saccadic adaptation, error information unconsciously changes the endpoint of a saccade prompted by a visual target that shifts its final position during the saccade. We recorded CS from PC of the posterior vermis before, during, and after saccadic adaptation. In clear contradiction to the "error signal" concept, we found that CS occurred at random before adaptation onset, i.e., when the error was maximal, and built up to a specific saccade-related discharge profile during the course of adaptation. This profile became most pronounced at the end of adaptation, i.e., when the error had been nulled. CONCLUSIONS: We suggest that CS firing may underlie the stabilization of a learned motor behavior, rather than serving as an electrophysiological correlate of an error.     

Multiple Motor Learning Strategies in Visuomotor Rotation

Background

When exposed to a continuous directional discrepancy between movements of a visible hand cursor and the actual hand (visuomotor rotation), subjects adapt their reaching movements so that the cursor is brought to the target. Abrupt removal of the discrepancy after training induces reaching error in the direction opposite to the original discrepancy, which is called an aftereffect. Previous studies have shown that training with gradually increasing visuomotor rotation results in a larger aftereffect than with a suddenly increasing one. Although the aftereffect difference implies a difference in the learning process, it is still unclear whether the learned visuomotor transformations are qualitatively different between the training conditions.

Methodology/Principal Findings

We examined the qualitative changes in the visuomotor transformation after the learning of the sudden and gradual visuomotor rotations. The learning of the sudden rotation led to a significant increase of the reaction time for arm movement initiation and then the reaching error decreased, indicating that the learning is associated with an increase of computational load in motor preparation (planning). In contrast, the learning of the gradual rotation did not change the reaction time but resulted in an increase of the gain of feedback control, suggesting that the online adjustment of the reaching contributes to the learning of the gradual rotation. When the online cursor feedback was eliminated during the learning of the gradual rotation, the reaction time increased, indicating that additional computations are involved in the learning of the gradual rotation.

Conclusions/Significance

The results suggest that the change in the motor planning and online feedback adjustment of the movement are involved in the learning of the visuomotor rotation. The contributions of those computations to the learning are flexibly modulated according to the visual environment. Such multiple learning strategies would be required for reaching adaptation within a short training period.     

Visuomotor Adaptation: How Forgetting Keeps Us Conservative

Even when provided with feedback after every movement, adaptation levels off before biases are completely removed. Incomplete adaptation has recently been attributed to forgetting: the adaptation is already partially forgotten by the time the next movement is made. Here we test whether this idea is correct. If so, the final level of adaptation is determined by a balance between learning and forgetting. Because we learn from perceived errors, scaling these errors by a magnification factor has the same effect as subjects increasing the amount by which they learn from each error. In contrast, there is no reason to expect scaling the errors to affect forgetting. The magnification factor should therefore influence the balance between learning and forgetting, and thereby the final level of adaptation. We found that adaptation was indeed more complete for larger magnification factors. This supports the idea that incomplete adaptation is caused by part of what has been learnt quickly being forgotten.     

Modelling the learning of biomechanics and visual planning for decision-making of motor actions

Recent experiments showed that the bio-mechanical ease and end-point stability associated to reaching movements are predicted prior to movement onset, and that these factors exert a significant influence on the choice of movement. As an extension of these results, here we investigate whether the knowledge about biomechanical costs and their influence on decision-making are the result of an adaptation process taking place during each experimental session or whether this knowledge was learned at an earlier stage of development. Specifically, we analysed both the pattern of decision-making and its fluctuations during each session, of several human subjects making free choices between two reaching movements that varied in path distance (target relative distance), biomechanical cost, aiming accuracy and stopping requirement. Our main result shows that the effect of biomechanics is well established at the start of the session, and that, consequently, the learning of biomechanical costs in decision-making occurred at an earlier stage of development. As a means to characterise the dynamics of this learning process, we also developed a model-based reinforcement learning model, which generates a possible account of how biomechanics may be incorporated into the motor plan to select between reaching movements. Results obtained in simulation showed that, after some pre-training corresponding to a motor babbling phase, the model can reproduce the subjects’ overall movement preferences. Although preliminary, this supports that the knowledge about biomechanical costs may have been learned in this manner, and supports the hypothesis that the fluctuations observed in the subjects’ behaviour may adapt in a similar fashion.     

Interference in ballistic motor learning: specificity and role of sensory error signals

Humans are capable of learning numerous motor skills, but newly acquired skills may be abolished by subsequent learning. Here we ask what factors determine whether interference occurs in motor learning. We speculated that interference requires competing processes of synaptic plasticity in overlapping circuits and predicted specificity. To test this, subjects learned a ballistic motor task. Interference was observed following subsequent learning of an accuracy-tracking task, but only if the competing task involved the same muscles and movement direction. Interference was not observed from a non-learning task suggesting that interference requires competing learning. Subsequent learning of the competing task 4 h after initial learning did not cause interference suggesting disruption of early motor memory consolidation as one possible mechanism underlying interference. Repeated transcranial magnetic stimulation (rTMS) of corticospinal motor output at intensities below movement threshold did not cause interference, whereas suprathreshold rTMS evoking motor responses and (re)afferent activation did. Finally, the experiments revealed that suprathreshold repetitive electrical stimulation of the agonist (but not antagonist) peripheral nerve caused interference. The present study is, to our knowledge, the first to demonstrate that peripheral nerve stimulation may cause interference. The finding underscores the importance of sensory feedback as error signals in motor learning. We conclude that interference requires competing plasticity in overlapping circuits. Interference is remarkably specific for circuits involved in a specific movement and it may relate to sensory error signals.     

Visual feedback is not necessary for the learning of novel dynamics

Background

When learning to perform a novel sensorimotor task, humans integrate multi-modal sensory feedback such as vision and proprioception in order to make the appropriate adjustments to successfully complete the task. Sensory feedback is used both during movement to control and correct the current movement, and to update the feed-forward motor command for subsequent movements. Previous work has shown that adaptation to stable dynamics is possible without visual feedback. However, it is not clear to what degree visual information during movement contributes to this learning or whether it is essential to the development of an internal model or impedance controller.

Methodology/Principle Findings

We examined the effects of the removal of visual feedback during movement on the learning of both stable and unstable dynamics in comparison with the case when both vision and proprioception are available. Subjects were able to learn to make smooth movements in both types of novel dynamics after learning with or without visual feedback. By examining the endpoint stiffness and force after learning it could be shown that subjects adapted to both types of dynamics in the same way whether they were provided with visual feedback of their trajectory or not. The main effects of visual feedback were to increase the success rate of movements, slightly straighten the path, and significantly reduce variability near the end of the movement.

Conclusions/Significance

These findings suggest that visual feedback of the hand during movement is not necessary for the adaptation to either stable or unstable novel dynamics. Instead vision appears to be used to fine-tune corrections of hand trajectory at the end of reaching movements.     

The binding of learning to action in motor adaptation

In motor tasks, errors between planned and actual movements generally result in adaptive changes which reduce the occurrence of similar errors in the future. It has commonly been assumed that the motor adaptation arising from an error occurring on a particular movement is specifically associated with the motion that was planned. Here we show that this is not the case. Instead, we demonstrate the binding of the adaptation arising from an error on a particular trial to the motion experienced on that same trial. The formation of this association means that future movements planned to resemble the motion experienced on a given trial benefit maximally from the adaptation arising from it. This reflects the idea that actual rather than planned motions are assigned 'credit' for motor errors because, in a computational sense, the maximal adaptive response would be associated with the condition credited with the error. We studied this process by examining the patterns of generalization associated with motor adaptation to novel dynamic environments during reaching arm movements in humans. We found that these patterns consistently matched those predicted by adaptation associated with the actual rather than the planned motion, with maximal generalization observed where actual motions were clustered. We followed up these findings by showing that a novel training procedure designed to leverage this newfound understanding of the binding of learning to action, can improve adaptation rates by greater than 50%. Our results provide a mechanistic framework for understanding the effects of partial assistance and error augmentation during neurologic rehabilitation, and they suggest ways to optimize their use.     

Saccadic motor planning by integrating visual information and pre-information on neural dynamic fields

A functional model of target selection in the saccadic system is presented, incorporating elements of visual processing, motor planning, and motor control. We address the integration of visual information with pre-information. which is provided by manipulating the probability that a target appears at a certain location. This integration is achieved within a dynamic representation of planned eye movement which is modeled through distributions of activation on a topographic field. Visual input evokes activation, which is also constrained by lateral interaction within the field and by preshaping input representing pre-information. The model describes target selection observable in paradigms in which visual goals are presented at more than one location. Specifically, we model the transition from averaging, where endpoints of first saccades fall between two visual target locations, to decision making, where endpoints of first saccades fall accurately onto one of two simultaneously presented visual targets. We make predictions about how metrical biases of first saccades are induced by pre-information about target locations acquired by learning. When coupled to a motor control stage, activation dynamics on the planning level contribute to stabilizing gaze under fixation conditions. The neurophysiological relevance of our functional model is discussed.     

Adaptive dynamic programming as a theory of sensorimotor control

Many characteristics of sensorimotor control can be explained by models based on optimization and optimal control theories. However, most of the previous models assume that the central nervous system has access to the precise knowledge of the sensorimotor system and its interacting environment. This viewpoint is difficult to be justified theoretically and has not been convincingly validated by experiments. To address this problem, this paper presents a new computational mechanism for sensorimotor control from a perspective of adaptive dynamic programming (ADP), which shares some features of reinforcement learning. The ADP-based model for sensorimotor control suggests that a command signal for the human movement is derived directly from the real-time sensory data, without the need to identify the system dynamics. An iterative learning scheme based on the proposed ADP theory is developed, along with rigorous convergence analysis. Interestingly, the computational model as advocated here is able to reproduce the motor learning behavior observed in experiments where a divergent force field or velocity-dependent force field was present. In addition, this modeling strategy provides a clear way to perform stability analysis of the overall system. Hence, we conjecture that human sensorimotor systems use an ADP-type mechanism to control movements and to achieve successful adaptation to uncertainties present in the environment.     

Dynamics of Dual Prism Adaptation: Relating Novel Experimental Results to a Minimalistic Neural Model

In everyday life, humans interact with a dynamic environment often requiring rapid adaptation of visual perception and motor control. In particular, new visuo–motor mappings must be learned while old skills have to be kept, such that after adaptation, subjects may be able to quickly change between two different modes of generating movements (‘dual–adaptation’). A fundamental question is how the adaptation schedule determines the acquisition speed of new skills. Given a fixed number of movements in two different environments, will dual–adaptation be faster if switches (‘phase changes’) between the environments occur more frequently? We investigated the dynamics of dual–adaptation under different training schedules in a virtual pointing experiment. Surprisingly, we found that acquisition speed of dual visuo–motor mappings in a pointing task is largely independent of the number of phase changes. Next, we studied the neuronal mechanisms underlying this result and other key phenomena of dual–adaptation by relating model simulations to experimental data. We propose a simple and yet biologically plausible neural model consisting of a spatial mapping from an input layer to a pointing angle which is subjected to a global gain modulation. Adaptation is performed by reinforcement learning on the model parameters. Despite its simplicity, the model provides a unifying account for a broad range of experimental data: It quantitatively reproduced the learning rates in dual–adaptation experiments for both direct effect, i.e. adaptation to prisms, and aftereffect, i.e. behavior after removal of prisms, and their independence on the number of phase changes. Several other phenomena, e.g. initial pointing errors that are far smaller than the induced optical shift, were also captured. Moreover, the underlying mechanisms, a local adaptation of a spatial mapping and a global adaptation of a gain factor, explained asymmetric spatial transfer and generalization of prism adaptation, as observed in other experiments.     

Adaptive control of movement deceleration during saccades

As you read this text, your eyes make saccades that guide your fovea from one word to the next. Accuracy of these movements require the brain to monitor and learn from visual errors. A current model suggests that learning is supported by two different adaptive processes, one fast (high error sensitivity, low retention), and the other slow (low error sensitivity, high retention). Here, we searched for signatures of these hypothesized processes and found that following experience of a visual error, there was an adaptive change in the motor commands of the subsequent saccade. Surprisingly, this adaptation was not uniformly expressed throughout the movement. Rather, after experience of a single error, the adaptive response in the subsequent trial was limited to the deceleration period. After repeated exposure to the same error, the acceleration period commands also adapted, and exhibited resistance to forgetting during set-breaks. In contrast, the deceleration period commands adapted more rapidly, but suffered from poor retention during these same breaks. State-space models suggested that acceleration and deceleration periods were supported by a shared adaptive state which re-aimed the saccade, as well as two separate processes which resembled a two-state model: one that learned slowly and contributed primarily via acceleration period commands, and another that learned rapidly but contributed primarily via deceleration period commands.