Rethinking motor learning and savings in adaptation paradigms: model-free memory for successful actions combines with internal models

Although motor learning is likely to involve multiple processes, phenomena observed in error-based motor learning paradigms tend to be conceptualized in terms of only a single process: adaptation, which occurs through updating an internal model. Here we argue that fundamental phenomena like movement direction biases, savings (faster relearning), and interference do not relate to adaptation but instead are attributable to two additional learning processes that can be characterized as model-free: use-dependent plasticity and operant reinforcement. Although usually "hidden" behind adaptation, we demonstrate, with modified visuomotor rotation paradigms, that these distinct model-based and model-free processes combine to learn an error-based motor task. (1) Adaptation of an internal model channels movements toward successful error reduction in visual space. (2) Repetition of the newly adapted movement induces directional biases toward the?repeated movement. (3) Operant reinforcement through association of the adapted movement with successful error reduction is responsible for savings.     

Adaptive control of movement deceleration during saccades

As you read this text, your eyes make saccades that guide your fovea from one word to the next. Accuracy of these movements require the brain to monitor and learn from visual errors. A current model suggests that learning is supported by two different adaptive processes, one fast (high error sensitivity, low retention), and the other slow (low error sensitivity, high retention). Here, we searched for signatures of these hypothesized processes and found that following experience of a visual error, there was an adaptive change in the motor commands of the subsequent saccade. Surprisingly, this adaptation was not uniformly expressed throughout the movement. Rather, after experience of a single error, the adaptive response in the subsequent trial was limited to the deceleration period. After repeated exposure to the same error, the acceleration period commands also adapted, and exhibited resistance to forgetting during set-breaks. In contrast, the deceleration period commands adapted more rapidly, but suffered from poor retention during these same breaks. State-space models suggested that acceleration and deceleration periods were supported by a shared adaptive state which re-aimed the saccade, as well as two separate processes which resembled a two-state model: one that learned slowly and contributed primarily via acceleration period commands, and another that learned rapidly but contributed primarily via deceleration period commands.     

Reduction in Learning Rates Associated with Anterograde Interference Results from Interactions between Different Timescales in Motor Adaptation

Prior experiences can influence future actions. These experiences can not only drive adaptive changes in motor output, but they can also modulate the rate at which these adaptive changes occur. Here we studied anterograde interference in motor adaptation – the ability of a previously learned motor task (Task A) to reduce the rate of subsequently learning a different (and usually opposite) motor task (Task B). We examined the formation of the motor system''s capacity for anterograde interference in the adaptive control of human reaching-arm movements by determining the amount of interference after varying durations of exposure to Task A (13, 41, 112, 230, and 369 trials). We found that the amount of anterograde interference observed in the learning of Task B increased with the duration of Task A. However, this increase did not continue indefinitely; instead, the interference reached asymptote after 15–40 trials of Task A. Interestingly, we found that a recently proposed multi-rate model of motor adaptation, composed of two distinct but interacting adaptive processes, predicts several key features of the interference patterns we observed. Specifically, this computational model (without any free parameters) predicts the initial growth and leveling off of anterograde interference that we describe, as well as the asymptotic amount of interference that we observe experimentally (R² = 0.91). Understanding the mechanisms underlying anterograde interference in motor adaptation may enable the development of improved training and rehabilitation paradigms that mitigate unwanted interference.     

Sensory processing of motor inaccuracy depends on previously performed movement and on subsequent motor corrections: a study of the saccadic system

When goal-directed movements are inaccurate, two responses are generated by the brain: a fast motor correction toward the target and an adaptive motor recalibration developing progressively across subsequent trials. For the saccadic system, there is a clear dissociation between the fast motor correction (corrective saccade production) and the adaptive motor recalibration (primary saccade modification). Error signals used to trigger corrective saccades and to induce adaptation are based on post-saccadic visual feedback. The goal of this study was to determine if similar or different error signals are involved in saccadic adaptation and in corrective saccade generation. Saccadic accuracy was experimentally altered by systematically displacing the visual target during motor execution. Post-saccadic error signals were studied by manipulating visual information in two ways. First, the duration of the displaced target after primary saccade termination was set at 15, 50, 100 or 800 ms in different adaptation sessions. Second, in some sessions, the displaced target was followed by a visual mask that interfered with visual processing. Because they rely on different mechanisms, the adaptation of reactive saccades and the adaptation of voluntary saccades were both evaluated. We found that saccadic adaptation and corrective saccade production were both affected by the manipulations of post-saccadic visual information, but in different ways. This first finding suggests that different types of error signal processing are involved in the induction of these two motor corrections. Interestingly, voluntary saccades required a longer duration of post-saccadic target presentation to reach the same amount of adaptation as reactive saccades. Finally, the visual mask interfered with the production of corrective saccades only during the voluntary saccades adaptation task. These last observations suggest that post-saccadic perception depends on the previously performed action and that the differences between saccade categories of motor correction and adaptation occur at an early level of visual processing.     

Time and tide in cerebellar memory formation

The notion that the olivocerebellar system is crucial for motor learning is well established. In recent years, it has become evident that there can be many forms of both synaptic and non-synaptic plasticity within this system and that each might have a different role in developing and maintaining motor learning across a wide range of tasks. There are several possible molecular and cellular mechanisms that could underlie adaptation of the vestibulo-ocular reflex and eyeblink conditioning. Although causal relationships between particular cellular processes and individual components of a learned behaviour have not been demonstrated unequivocally, an overall picture is emerging that the different types and sites of cellular plasticity relate importantly to the stage of learning and/or its temporal specifics.     

Learning from sensory and reward prediction errors during motor adaptation

Voluntary motor commands produce two kinds of consequences. Initially, a sensory consequence is observed in terms of activity in our primary sensory organs (e.g., vision, proprioception). Subsequently, the brain evaluates the sensory feedback and produces a subjective measure of utility or usefulness of the motor commands (e.g., reward). As a result, comparisons between predicted and observed consequences of motor commands produce two forms of prediction error. How do these errors contribute to changes in motor commands? Here, we considered a reach adaptation protocol and found that when high quality sensory feedback was available, adaptation of motor commands was driven almost exclusively by sensory prediction errors. This form of learning had a distinct signature: as motor commands adapted, the subjects altered their predictions regarding sensory consequences of motor commands, and generalized this learning broadly to neighboring motor commands. In contrast, as the quality of the sensory feedback degraded, adaptation of motor commands became more dependent on reward prediction errors. Reward prediction errors produced comparable changes in the motor commands, but produced no change in the predicted sensory consequences of motor commands, and generalized only locally. Because we found that there was a within subject correlation between generalization patterns and sensory remapping, it is plausible that during adaptation an individual''s relative reliance on sensory vs. reward prediction errors could be inferred. We suggest that while motor commands change because of sensory and reward prediction errors, only sensory prediction errors produce a change in the neural system that predicts sensory consequences of motor commands.     

Cerebellar complex spike firing is suitable to induce as well as to stabilize motor learning

BACKGROUND: Cerebellar Purkinje cells (PC) generate two responses: the simple spike (SS), with high firing rates (>100 Hz), and the complex spike (CS), characterized by conspicuously low discharge rates (1-2 Hz). Contemporary theories of cerebellar learning suggest that the CS discharge pattern encodes an error signal that drives changes in SS activity, ultimately related to motor behavior. This then predicts that CS will discharge in relation to the error and at random once the error has been nulled by the new behavior. RESULTS: We tested this hypothesis with saccadic adaptation in macaque monkeys as a model of cerebellar-dependent motor learning. During saccadic adaptation, error information unconsciously changes the endpoint of a saccade prompted by a visual target that shifts its final position during the saccade. We recorded CS from PC of the posterior vermis before, during, and after saccadic adaptation. In clear contradiction to the "error signal" concept, we found that CS occurred at random before adaptation onset, i.e., when the error was maximal, and built up to a specific saccade-related discharge profile during the course of adaptation. This profile became most pronounced at the end of adaptation, i.e., when the error had been nulled. CONCLUSIONS: We suggest that CS firing may underlie the stabilization of a learned motor behavior, rather than serving as an electrophysiological correlate of an error.     

Characterization of synaptically connected nuclei in a potential sensorimotor feedback pathway in the zebra finch song system

Birdsong is a learned behavior that is controlled by a group of identified nuclei, known collectively as the song system. The cortical nucleus HVC (used as a proper name) is a focal point of many investigations as it is necessary for song production, song learning, and receives selective auditory information. HVC receives input from several sources including the cortical area MMAN (medial magnocellular nucleus of the nidopallium). The MMAN to HVC connection is particularly interesting as it provides potential sensorimotor feedback to HVC. To begin to understand the role of this connection, we investigated the physiological relation between MMAN and HVC activity with simultaneous multiunit extracellular recordings from these two nuclei in urethane anesthetized zebra finches. As previously reported, we found similar timing in spontaneous bursts of activity in MMAN and HVC. Like HVC, MMAN responds to auditory playback of the bird's own song (BOS), but had little response to reversed BOS or conspecific song. Stimulation of MMAN resulted in evoked activity in HVC, indicating functional excitation from MMAN to HVC. However, inactivation of MMAN resulted in no consistent change in auditory responses in HVC. Taken together, these results indicate that MMAN provides functional excitatory input to HVC but does not provide significant auditory input to HVC in anesthetized animals. We hypothesize that MMAN may play a role in motor reinforcement or coordination, or may provide modulatory input to the song system about the internal state of the animal as it receives input from the hypothalamus.     

Estimating the relevance of world disturbances to explain savings, interference and long-term motor adaptation effects

Recent studies suggest that motor adaptation is the result of multiple, perhaps linear processes each with distinct time scales. While these models are consistent with some motor phenomena, they can neither explain the relatively fast re-adaptation after a long washout period, nor savings on a subsequent day. Here we examined if these effects can be explained if we assume that the CNS stores and retrieves movement parameters based on their possible relevance. We formalize this idea with a model that infers not only the sources of potential motor errors, but also their relevance to the current motor circumstances. In our model adaptation is the process of re-estimating parameters that represent the body and the world. The likelihood of a world parameter being relevant is then based on the mismatch between an observed movement and that predicted when not compensating for the estimated world disturbance. As such, adapting to large motor errors in a laboratory setting should alert subjects that disturbances are being imposed on them, even after motor performance has returned to baseline. Estimates of this external disturbance should be relevant both now and in future laboratory settings. Estimated properties of our bodies on the other hand should always be relevant. Our model demonstrates savings, interference, spontaneous rebound and differences between adaptation to sudden and gradual disturbances. We suggest that many issues concerning savings and interference can be understood when adaptation is conditioned on the relevance of parameters.     

Involvement of human basal ganglia in offline feedback control of voluntary movement

Practice makes perfect, but the neural substrates of trial-to-trial learning in motor tasks remain unclear. There is some evidence that the basal ganglia process feedback-related information to modify learning in essentially cognitive tasks , but the evidence that these key motor structures are involved in offline feedback-related improvement of performance in motor tasks is paradoxically limited. Lesion studies in adult zebra finches suggest that the avian basal ganglia are involved in the transmission or production of an error signal during song . However, patients with Huntington's disease, in which there is prominent basal ganglia dysfunction, are not impaired in error-dependent modulation of future trial performance . By directly recording from the subthalamic nucleus in patients with Parkinson's disease, we demonstrate that this nucleus processes error in trial performance at short latency. Local evoked activity is greatest in response to smallest errors and influences the programming of subsequent movements. Accordingly, motor parameters are least likely to change after the greatest evoked responses so that accurately performed trials tend to precede other accurate trials. This relationship is disrupted by electrical stimulation of the nucleus at high frequency. Thus, the human subthalamic nucleus is involved in feedback-based learning.     

Quantitative analysis of aspartate receptor signaling complex reveals that the homogeneous two-state model is inadequate: development of a heterogeneous two-state model

The two-state model of receptor activation, in which a receptor population exists in equilibrium between a single on-state and a single off-state, has long been considered a viable model for the signaling behavior of bacterial chemoreceptors. Here, we show that this simple, homogeneous two-state model is adequate for a pure receptor population with just one adaptation state, but fails to account quantitatively for the observed linear relationship between the apparent attractant affinity (K(1/2)) and kinase activity (V(obs)(apo)) as the adaptation state is varied. Further analysis reveals that the available data are instead consistent with a heterogeneous two-state model in which covalent modification of receptor adaptation sites changes the microscopic properties of the on-state or off-state. In such a system, each receptor molecule retains a single on-state and off-state, but covalent adaptation generates a heterogeneous population of receptors exhibiting a range of different on-states or off-states with different microscopic parameters and conformations. It follows that covalent adaptation transforms the receptor from a simple, two-state toggle switch into a variable switch. In order to identify the microscopic parameters most sensitive to covalent adaptation, six modified, two-state models were examined in which covalent adaptation alters a different microscopic parameter. The analysis suggests that covalent adaptation primarily alters the ligand-binding affinity of the receptor off-state (K(D1)). By contrast, models in which covalent adaptation alters the ligand-binding affinity of the receptor on-state, the maximal kinase stimulation of the on-state or off-state, cooperative interactions between receptors, or the assembly of the receptor-kinase signaling complex are inconsistent with the available evidence. Overall, the findings support a heterogeneous two-state model in which modification of the receptor adaptation sites generates a population of receptors with heterogeneous off-states differing in their attractant affinities.In the process of testing the effects of covalent adaptation on the assembly of the receptor-kinase signaling complex, a new method for estimating the stoichiometric ratio of receptor and CheA in the ternary signaling complex was devised. This method suggests that the ratio of receptor dimers to CheA dimers in the assembled complex is 6:1 or less.     

Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses

 Some characteristics of arm movements that humans exhibit during learning the dynamics of reaching are consistent with a theoretical framework where training results in motor commands that are gradually modified to predict and compensate for novel forces that may act on the hand. As a first approximation, the motor control system behaves as an adapting controller that learns an internal model of the dynamics of the task. It approximates inverse dynamics and predicts motor commands that are appropriate for a desired limb trajectory. However, we had previously noted that subtle motion characteristics observed during changes in task dynamics challenged this simple model and raised the possibility that adaptation also involved sensory–motor feedback pathways. These pathways reacted to sensory feedback during the course of the movement. Here we hypothesize that adaptation to dynamics might also involve a modification of how the CNS responds to sensory feedback. We tested this through experiments that quantified how the motor system's response to errors during voluntary movements changed as it adapted to dynamics of a force field. We describe a nonlinear approach that approximates the impedance of the arm, i.e., force response as a function of arm displacement trajectory. We observe that after adaptation, the impedance function changes in a way that closely matches and counters the effect of the force field. This is particularly prominent in the long-latency (>100 ms) component of response to perturbations. Therefore, it appears that practice not only modifies the internal model with which the brain generates motor commands that initiate a movement, but also the internal model with which sensory feedback is integrated with the ongoing descending commands in order to respond to error during the movement. Received: 10 January 2001 / Accepted in revised form: 30 May 2001     

Coordinates transformation and learning control for visually-guided voluntary movement with iteration: A Newton-like method in a function space

In order to control visually-guided voluntary movements, the central nervous system (CNS) must solve the following three computational problems at different levels: (1) determination of a desired trajectory in the visual coordinates, (2) transformation of the coordinates of the desired trajectory to the body coordinates and (3) generation of motor command. In this paper, the second and the third problems are treated at computational, representational and hardware levels of Marr. We first study the problems at the computational level, and then propose an iterative learning scheme as a possible algorithm. This is a trial and error type learning such as repetitive training of golf swing. The amount of motor command needed to coordinate activities of many muscles is not determined at once, but in a step-wise, trial and error fashion in the course of a set of repetitions. Actually, the motor command in the (n+1)-th iteration is a sum of the motor command in then-th iteration plus two modification terms which are, respectively, proportional to acceleration and speed errors between the desired trajectory and the realized trajectory in then-th iteration. We mathematically formulate this iterative learning control as a Newton-like method in functional spaces and prove its convergence under appropriate mathematical conditions with use of dynamical system theory and functional analysis. Computer simulations of this iterative learning control of a robotic manipulator in the body or visual coordinates are shown. Finally, we propose that areas 2, 5, and 7 of the sensory association cortex are possible sites of this learning control. Further we propose neural network model which acquires transformation matrices from acceleration or velocity to motor command, which are used in these schemes.     

Simultaneous Processing of Information on Multiple Errors in Visuomotor Learning

The proper association between planned and executed movements is crucial for motor learning because the discrepancies between them drive such learning. Our study explored how this association was determined when a single action caused the movements of multiple visual objects. Participants reached toward a target by moving a cursor, which represented the right hand’s position. Once every five to six normal trials, we interleaved either of two kinds of visual perturbation trials: rotation of the cursor by a certain amount (±15°, ±30°, and ±45°) around the starting position (single-cursor condition) or rotation of two cursors by different angles (+15° and −45°, 0° and 30°, etc.) that were presented simultaneously (double-cursor condition). We evaluated the aftereffects of each condition in the subsequent trial. The error sensitivity (ratio of the aftereffect to the imposed visual rotation) in the single-cursor trials decayed with the amount of rotation, indicating that the motor learning system relied to a greater extent on smaller errors. In the double-cursor trials, we obtained a coefficient that represented the degree to which each of the visual rotations contributed to the aftereffects based on the assumption that the observed aftereffects were a result of the weighted summation of the influences of the imposed visual rotations. The decaying pattern according to the amount of rotation was maintained in the coefficient of each imposed visual rotation in the double-cursor trials, but the value was reduced to approximately 40% of the corresponding error sensitivity in the single-cursor trials. We also found a further reduction of the coefficients when three distinct cursors were presented (e.g., −15°, 15°, and 30°). These results indicated that the motor learning system utilized multiple sources of visual error information simultaneously to correct subsequent movement and that a certain averaging mechanism might be at work in the utilization process.     

fMRI adaptation reveals mirror neurons in human inferior parietal cortex

Mirror neurons, as originally described in the macaque, have two defining properties [1, 2]: They respond specifically to a particular action (e.g., bringing an object to the mouth), and they produce their action-specific responses independent of whether the monkey executes the action or passively observes a conspecific performing the same action. In humans, action observation and action execution engage a network of frontal, parietal, and temporal areas. However, it is unclear whether these responses reflect the activity of a single population that represents both observed and executed actions in a common neural code or the activity of distinct but overlapping populations of exclusively perceptual and motor neurons [3]. Here, we used fMRI adaptation to show that the right inferior parietal lobe (IPL) responds independently to specific actions regardless of whether they are observed or executed. Specifically, responses in the right IPL were attenuated when participants observed a recently executed action relative to one that had not previously been performed. This adaptation across action and perception demonstrates that the right IPL responds selectively to the motoric and perceptual representations of actions and is the first evidence for a neural response in humans that shows both defining properties of mirror neurons.     

Spatiotopic visual maps revealed by saccadic adaptation in humans

Saccadic adaptation [1] is a powerful experimental paradigm to probe the mechanisms of eye movement control and spatial vision, in which saccadic amplitudes change in response to false visual feedback. The adaptation occurs primarily in the motor system [2, 3], but there is also evidence for visual adaptation, depending on the size and the permanence of the postsaccadic error [4-7]. Here we confirm that adaptation has a strong visual component and show that the visual component of the adaptation is spatially selective in external, not retinal coordinates. Subjects performed?a memory-guided, double-saccade, outward-adaptation task designed to maximize visual adaptation and to dissociate the visual and motor corrections. When the memorized saccadic target was in the same position (in external space) as that used in the adaptation training, saccade targeting was strongly influenced by adaptation (even if not matched in retinal or cranial position), but when in the same retinal or cranial but different external spatial position, targeting was unaffected by adaptation, demonstrating unequivocal spatiotopic selectivity. These results point to the existence of a spatiotopic neural representation for eye movement control that adapts in response to saccade error signals.     

Motor imagery—Anatomical representation and electrophysiological characteristics

Experience and results of neuropsychological studies have shown that motor imagery can improve motor performance and enhance motor learning. In recent years several electro-physiological and functional imaging studies have investigated the physiological basis for this observation. In the present essay we review two of our recent studies, in which we compared motor imagery with motor preparation and motor execution. In the first we used positron emission tomography to describe their functional anatomy and in the second we employed electromyography, H-reflexes and transcranial magnetic stimulation to delineate their electrophysiological characteristics. Both studies demonstrated that motor imagery shares some characteristics with motor preparation and other, additional ones with motor execution. Thus it can be seen as a special form of motor behaviour, similar but distinct from both motor preparation and execution. This combination of mutual and distinct characteristics may be the key to its successful role in motor learning. Special issue dedicated to Dr. Herman Bachelard.     

Compensation for changing motor uncertainty

When movement outcome differs consistently from the intended movement, errors are used to correct subsequent movements (e.g., adaptation to displacing prisms or force fields) by updating an internal model of motor and/or sensory systems. Here, we examine changes to an internal model of the motor system under changes in the variance structure of movement errors lacking an overall bias. We introduced a horizontal visuomotor perturbation to change the statistical distribution of movement errors anisotropically, while monetary gains/losses were awarded based on movement outcomes. We derive predictions for simulated movement planners, each differing in its internal model of the motor system. We find that humans optimally respond to the overall change in error magnitude, but ignore the anisotropy of the error distribution. Through comparison with simulated movement planners, we found that aimpoints corresponded quantitatively to an ideal movement planner that updates a strictly isotropic (circular) internal model of the error distribution. Aimpoints were planned in a manner that ignored the direction-dependence of error magnitudes, despite the continuous availability of unambiguous information regarding the anisotropic distribution of actual motor errors.     

Interrelated mechanisms in reward and learning

This brief review is focused on recent work in our laboratory, in which we assayed nicotine-induced neurotransmitter changes, comparing it to changes induced by other compounds and examined the receptor systems and their interactions that mediate the changes. The primary aim of our studies is to examine the role of neurotransmitter changes in reward and learning processes. We find that these processes are interlinked and interact in that reward-addiction mechanisms include processes of learning and learning-memory mechanisms include processes of reward. In spite being interlinked, the two processes have different functions and distinct properties and our long-term aim is to identify factors that control these processes and the differences among the processes. Here, we discuss reward processes, which we define as changes examined after administration of nicotine, cocaine or food, each of which induces changes in neurotransmitter levels and functions in cognitive areas as well as in reward areas. The changes are regionally heterogeneous and are drug or stimulus specific. They include changes in the transmitters assayed (catecholamines, amino acids, and acetylcholine) and also in their metabolites, hence, in addition to release, uptake and metabolism are involved. Many receptors modulate the response with direct and indirect effects. The involvement of many transmitters, receptors and their interactions and the stimulus specificity of the response indicated that each function, reward and learning represents the involvement of different pattern of changes with a different stimulus, therefore, many different learning and many different reward processes are active, which allow stimulus specific responses. The complex pattern of reward-induced changes in neurotransmitters is only a part of the multiple changes observed, but one which has a crucial and controlling function.     

Reexposure to a sensorimotor perturbation produces opposite effects on explicit and implicit learning processes

The motor system demonstrates an exquisite ability to adapt to changes in the environment and to quickly reset when these changes prove transient. If similar environmental changes are encountered in the future, learning may be faster, a phenomenon known as savings. In studies of sensorimotor learning, a central component of savings is attributed to the explicit recall of the task structure and appropriate compensatory strategies. Whether implicit adaptation also contributes to savings remains subject to debate. We tackled this question by measuring, in parallel, explicit and implicit adaptive responses in a visuomotor rotation task, employing a protocol that typically elicits savings. While the initial rate of learning was faster in the second exposure to the perturbation, an analysis decomposing the 2 processes showed the benefit to be solely associated with explicit re-aiming. Surprisingly, we found a significant decrease after relearning in aftereffect magnitudes during no-feedback trials, a direct measure of implicit adaptation. In a second experiment, we isolated implicit adaptation using clamped visual feedback, a method known to eliminate the contribution of explicit learning processes. Consistent with the results of the first experiment, participants exhibited a marked reduction in the adaptation function, as well as an attenuated aftereffect when relearning from the clamped feedback. Motivated by these results, we reanalyzed data from prior studies and observed a consistent, yet unappreciated pattern of attenuation of implicit adaptation during relearning. These results indicate that explicit and implicit sensorimotor processes exhibit opposite effects upon relearning: Explicit learning shows savings, while implicit adaptation becomes attenuated

Humans learning a new motor task typically improve with repeated practice due to the faster expression of more effective explicit strategies; this study reveals that when motor learning occurs without awareness, performance deteriorates upon relearning.