Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses

 Some characteristics of arm movements that humans exhibit during learning the dynamics of reaching are consistent with a theoretical framework where training results in motor commands that are gradually modified to predict and compensate for novel forces that may act on the hand. As a first approximation, the motor control system behaves as an adapting controller that learns an internal model of the dynamics of the task. It approximates inverse dynamics and predicts motor commands that are appropriate for a desired limb trajectory. However, we had previously noted that subtle motion characteristics observed during changes in task dynamics challenged this simple model and raised the possibility that adaptation also involved sensory–motor feedback pathways. These pathways reacted to sensory feedback during the course of the movement. Here we hypothesize that adaptation to dynamics might also involve a modification of how the CNS responds to sensory feedback. We tested this through experiments that quantified how the motor system's response to errors during voluntary movements changed as it adapted to dynamics of a force field. We describe a nonlinear approach that approximates the impedance of the arm, i.e., force response as a function of arm displacement trajectory. We observe that after adaptation, the impedance function changes in a way that closely matches and counters the effect of the force field. This is particularly prominent in the long-latency (>100 ms) component of response to perturbations. Therefore, it appears that practice not only modifies the internal model with which the brain generates motor commands that initiate a movement, but also the internal model with which sensory feedback is integrated with the ongoing descending commands in order to respond to error during the movement. Received: 10 January 2001 / Accepted in revised form: 30 May 2001     

Adaptation to visual feedback delay influences visuomotor learning

Computational theory of motor control suggests that the brain continuously monitors motor commands, to predict their sensory consequences before actual sensory feedback becomes available. Such prediction error is a driving force of motor learning, and therefore appropriate associations between motor commands and delayed sensory feedback signals are crucial. Indeed, artificially introduced delays in visual feedback have been reported to degrade motor learning. However, considering our perceptual ability to causally bind our own actions with sensory feedback, demonstrated by the decrease in the perceived time delay following repeated exposure to an artificial delay, we hypothesized that such perceptual binding might alleviate deficits of motor learning associated with delayed visual feedback. Here, we evaluated this hypothesis by investigating the ability of human participants to adapt their reaching movements in response to a novel visuomotor environment with 3 visual feedback conditions--no-delay, sudden-delay, and adapted-delay. To introduce novelty into the trials, the cursor position, which originally indicated the hand position in baseline trials, was rotated around the starting position. In contrast to the no-delay condition, a 200-ms delay was artificially introduced between the cursor and hand positions during the presence of visual rotation (sudden-delay condition), or before the application of visual rotation (adapted-delay condition). We compared the learning rate (representing how the movement error modifies the movement direction in the subsequent trial) between the 3 conditions. In comparison with the no-delay condition, the learning rate was significantly degraded for the sudden-delay condition. However, this degradation was significantly alleviated by prior exposure to the delay (adapted-delay condition). Our data indicate the importance of appropriate temporal associations between motor commands and sensory feedback in visuomotor learning. Moreover, they suggest that the brain is able to account for such temporal associations in a flexible manner.     

The sensory consequences of speaking: parametric neural cancellation during speech in auditory cortex

When we speak, we provide ourselves with auditory speech input. Efficient monitoring of speech is often hypothesized to depend on matching the predicted sensory consequences from internal motor commands (forward model) with actual sensory feedback. In this paper we tested the forward model hypothesis using functional Magnetic Resonance Imaging. We administered an overt picture naming task in which we parametrically reduced the quality of verbal feedback by noise masking. Presentation of the same auditory input in the absence of overt speech served as listening control condition. Our results suggest that a match between predicted and actual sensory feedback results in inhibition of cancellation of auditory activity because speaking with normal unmasked feedback reduced activity in the auditory cortex compared to listening control conditions. Moreover, during self-generated speech, activation in auditory cortex increased as the feedback quality of the self-generated speech decreased. We conclude that during speaking early auditory cortex is involved in matching external signals with an internally generated model or prediction of sensory consequences, the locus of which may reside in auditory or higher order brain areas. Matching at early auditory cortex may provide a very sensitive monitoring mechanism that highlights speech production errors at very early levels of processing and may efficiently determine the self-agency of speech input.     

Sensorimotor Recalibration Depends on Attribution of Sensory Prediction Errors to Internal Causes

Sensorimotor learning critically depends on error signals. Learning usually tries to minimise these error signals to guarantee optimal performance. Errors can, however, have both internal causes, resulting from one’s sensorimotor system, and external causes, resulting from external disturbances. Does learning take into account the perceived cause of error information? Here, we investigated the recalibration of internal predictions about the sensory consequences of one’s actions. Since these predictions underlie the distinction of self- and externally produced sensory events, we assumed them to be recalibrated only by prediction errors attributed to internal causes. When subjects were confronted with experimentally induced visual prediction errors about their pointing movements in virtual reality, they recalibrated the predicted visual consequences of their movements. Recalibration was not proportional to the externally generated prediction error, but correlated with the error component which subjects attributed to internal causes. We also revealed adaptation in subjects’ motor performance which reflected their recalibrated sensory predictions. Thus, causal attribution of error information is essential for sensorimotor learning.     

Prediction precedes control in motor learning

Skilled motor behavior relies on the brain learning both to control the body and predict the consequences of this control. Prediction turns motor commands into expected sensory consequences, whereas control turns desired consequences into motor commands. To capture this symmetry, the neural processes underlying prediction and control are termed the forward and inverse internal models, respectively. Here, we investigate how these two fundamental processes are related during motor learning. We used an object manipulation task in which subjects learned to move a hand-held object with novel dynamic properties along a prescribed path. We independently and simultaneously measured subjects' ability to control their actions and to predict their consequences. We found different time courses for predictor and controller learning, with prediction being learned far more rapidly than control. In early stages of manipulating the object, subjects could predict the consequences of their actions, as measured by the grip force they used to grasp the object, but could not generate appropriate actions for control, as measured by their hand trajectory. As predicted by several recent theoretical models of sensorimotor control, our results indicate that people can learn to predict the consequences of their actions before they can learn to control their actions.     

Sensorimotor Learning Biases Choice Behavior: A Learning Neural Field Model for Decision Making

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject''s learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.     

A symbiotic brain-machine interface through value-based decision making

Background

In the development of Brain Machine Interfaces (BMIs), there is a great need to enable users to interact with changing environments during the activities of daily life. It is expected that the number and scope of the learning tasks encountered during interaction with the environment as well as the pattern of brain activity will vary over time. These conditions, in addition to neural reorganization, pose a challenge to decoding neural commands for BMIs. We have developed a new BMI framework in which a computational agent symbiotically decoded users'' intended actions by utilizing both motor commands and goal information directly from the brain through a continuous Perception-Action-Reward Cycle (PARC).

Methodology

The control architecture designed was based on Actor-Critic learning, which is a PARC-based reinforcement learning method. Our neurophysiology studies in rat models suggested that Nucleus Accumbens (NAcc) contained a rich representation of goal information in terms of predicting the probability of earning reward and it could be translated into an evaluative feedback for adaptation of the decoder with high precision. Simulated neural control experiments showed that the system was able to maintain high performance in decoding neural motor commands during novel tasks or in the presence of reorganization in the neural input. We then implanted a dual micro-wire array in the primary motor cortex (M1) and the NAcc of rat brain and implemented a full closed-loop system in which robot actions were decoded from the single unit activity in M1 based on an evaluative feedback that was estimated from NAcc.

Conclusions

Our results suggest that adapting the BMI decoder with an evaluative feedback that is directly extracted from the brain is a possible solution to the problem of operating BMIs in changing environments with dynamic neural signals. During closed-loop control, the agent was able to solve a reaching task by capturing the action and reward interdependency in the brain.     

Learning to predict the future: the cerebellum adapts feedforward movement control

The role of the cerebellum in motor control and learning has been largely inferred from the effects of cerebellar damage. Recent work shows that cerebellar damage produces greater impairment of movements that require predictive as opposed to reactive control. This dissociation is consistent across many different types of movement. Predictive control is crucial for fast and ballistic movements, but impaired prediction can also affect slow movements, because of increased reliance on time-delayed feedback signals. The new findings are compatible with theories of cerebellar function, but still do not resolve whether the cerebellum operates by predicting the optimal motor commands or future sensory states. Prediction mechanisms must be learned and maintained through comparisons between predicted and observed outcomes. New results show that not all such error information is equivalent in driving cerebellar learning.     

Remapping auditory-motor representations in voice production

Evidence regarding visually guided limb movements suggests that the motor system learns and maintains neural maps between motor commands and sensory feedback. Such systems are hypothesized to be used in a feed-forward control strategy that permits precision and stability without the delays of direct feedback control. Human vocalizations involve precise control over vocal and respiratory muscles. However, little is known about the sensorimotor representations underlying speech production. Here, we manipulated the heard fundamental frequency of the voice during speech to demonstrate learning of auditory-motor maps. Mandarin speakers repeatedly produced words with specific pitch patterns (tone categories). On each successive utterance, the frequency of their auditory feedback was increased by 1/100 of a semitone until they heard their feedback one full semitone above their true pitch. Subjects automatically compensated for these changes by lowering their vocal pitch. When feedback was unexpectedly returned to normal, speakers significantly increased the pitch of their productions beyond their initial baseline frequency. This adaptation was found to generalize to the production of another tone category. However, results indicate that a more robust adaptation was produced for the tone that was spoken during feedback alteration. The immediate aftereffects suggest a global remapping of the auditory-motor relationship after an extremely brief training period. However, this learning does not represent a complete transformation of the mapping; rather, it is in part target dependent.     

Interference and Shaping in Sensorimotor Adaptations with Rewards

When a perturbation is applied in a sensorimotor transformation task, subjects can adapt and maintain performance by either relying on sensory feedback, or, in the absence of such feedback, on information provided by rewards. For example, in a classical rotation task where movement endpoints must be rotated to reach a fixed target, human subjects can successfully adapt their reaching movements solely on the basis of binary rewards, although this proves much more difficult than with visual feedback. Here, we investigate such a reward-driven sensorimotor adaptation process in a minimal computational model of the task. The key assumption of the model is that synaptic plasticity is gated by the reward. We study how the learning dynamics depend on the target size, the movement variability, the rotation angle and the number of targets. We show that when the movement is perturbed for multiple targets, the adaptation process for the different targets can interfere destructively or constructively depending on the similarities between the sensory stimuli (the targets) and the overlap in their neuronal representations. Destructive interferences can result in a drastic slowdown of the adaptation. As a result of interference, the time to adapt varies non-linearly with the number of targets. Our analysis shows that these interferences are weaker if the reward varies smoothly with the subject''s performance instead of being binary. We demonstrate how shaping the reward or shaping the task can accelerate the adaptation dramatically by reducing the destructive interferences. We argue that experimentally investigating the dynamics of reward-driven sensorimotor adaptation for more than one sensory stimulus can shed light on the underlying learning rules.     

An Adapting Auditory-motor Feedback Loop Can Contribute to Generating Vocal Repetition

Consecutive repetition of actions is common in behavioral sequences. Although integration of sensory feedback with internal motor programs is important for sequence generation, if and how feedback contributes to repetitive actions is poorly understood. Here we study how auditory feedback contributes to generating repetitive syllable sequences in songbirds. We propose that auditory signals provide positive feedback to ongoing motor commands, but this influence decays as feedback weakens from response adaptation during syllable repetitions. Computational models show that this mechanism explains repeat distributions observed in Bengalese finch song. We experimentally confirmed two predictions of this mechanism in Bengalese finches: removal of auditory feedback by deafening reduces syllable repetitions; and neural responses to auditory playback of repeated syllable sequences gradually adapt in sensory-motor nucleus HVC. Together, our results implicate a positive auditory-feedback loop with adaptation in generating repetitive vocalizations, and suggest sensory adaptation is important for feedback control of motor sequences.     

A Lightweight,Headphones-based System for Manipulating Auditory Feedback in Songbirds

Experimental manipulations of sensory feedback during complex behavior have provided valuable insights into the computations underlying motor control and sensorimotor plasticity¹. Consistent sensory perturbations result in compensatory changes in motor output, reflecting changes in feedforward motor control that reduce the experienced feedback error. By quantifying how different sensory feedback errors affect human behavior, prior studies have explored how visual signals are used to recalibrate arm movements^2,3 and auditory feedback is used to modify speech production^4-7. The strength of this approach rests on the ability to mimic naturalistic errors in behavior, allowing the experimenter to observe how experienced errors in production are used to recalibrate motor output.Songbirds provide an excellent animal model for investigating the neural basis of sensorimotor control and plasticity^8,9. The songbird brain provides a well-defined circuit in which the areas necessary for song learning are spatially separated from those required for song production, and neural recording and lesion studies have made significant advances in understanding how different brain areas contribute to vocal behavior^9-12. However, the lack of a naturalistic error-correction paradigm - in which a known acoustic parameter is perturbed by the experimenter and then corrected by the songbird - has made it difficult to understand the computations underlying vocal learning or how different elements of the neural circuit contribute to the correction of vocal errors¹³.The technique described here gives the experimenter precise control over auditory feedback errors in singing birds, allowing the introduction of arbitrary sensory errors that can be used to drive vocal learning. Online sound-processing equipment is used to introduce a known perturbation to the acoustics of song, and a miniaturized headphones apparatus is used to replace a songbird''s natural auditory feedback with the perturbed signal in real time. We have used this paradigm to perturb the fundamental frequency (pitch) of auditory feedback in adult songbirds, providing the first demonstration that adult birds maintain vocal performance using error correction¹⁴. The present protocol can be used to implement a wide range of sensory feedback perturbations (including but not limited to pitch shifts) to investigate the computational and neurophysiological basis of vocal learning.     

Adaptive properties of differential learning rates for positive and negative outcomes

The concept of the reward prediction error—the difference between reward obtained and reward predicted—continues to be a focal point for much theoretical and experimental work in psychology, cognitive science, and neuroscience. Models that rely on reward prediction errors typically assume a single learning rate for positive and negative prediction errors. However, behavioral data indicate that better-than-expected and worse-than-expected outcomes often do not have symmetric impacts on learning and decision-making. Furthermore, distinct circuits within cortico-striatal loops appear to support learning from positive and negative prediction errors, respectively. Such differential learning rates would be expected to lead to biased reward predictions and therefore suboptimal choice performance. Contrary to this intuition, we show that on static “bandit” choice tasks, differential learning rates can be adaptive. This occurs because asymmetric learning enables a better separation of learned reward probabilities. We show analytically how the optimal learning rate asymmetry depends on the reward distribution and implement a biologically plausible algorithm that adapts the balance of positive and negative learning rates from experience. These results suggest specific adaptive advantages for separate, differential learning rates in simple reinforcement learning settings and provide a novel, normative perspective on the interpretation of associated neural data.     

Illusory sensation of movement induced by repetitive transcranial magnetic stimulation

Human movement sense relies on both somatosensory feedback and on knowledge of the motor commands used to produce the movement. We have induced a movement illusion using repetitive transcranial magnetic stimulation over primary motor cortex and dorsal premotor cortex in the absence of limb movement and its associated somatosensory feedback. Afferent and efferent neural signalling was abolished in the arm with ischemic nerve block, and in the leg with spinal nerve block. Movement sensation was assessed following trains of high-frequency repetitive transcranial magnetic stimulation applied over primary motor cortex, dorsal premotor cortex, and a control area (posterior parietal cortex). Magnetic stimulation over primary motor cortex and dorsal premotor cortex produced a movement sensation that was significantly greater than stimulation over the control region. Movement sensation after dorsal premotor cortex stimulation was less affected by sensory and motor deprivation than was primary motor cortex stimulation. We propose that repetitive transcranial magnetic stimulation over dorsal premotor cortex produces a corollary discharge that is perceived as movement.     

When Money Is Not Enough: Awareness,Success, and Variability in Motor Learning

When performing a skill such as throwing a dart, many different combinations of joint motions suffice to hit the target. The motor system adapts rapidly to reduce bias in the desired outcome (i.e., the first-order moment of the error); however, the essence of skill is to produce movements with less variability (i.e., to reduce the second-order moment). It is easy to see how feedback about success or failure could sculpt performance to achieve this aim. However, it is unclear whether the dimensions responsible for success or failure need to be known explicitly by the subjects, or whether learning can proceed without explicit awareness of the movement parameters that need to change. Here, we designed a redundant, two-dimensional reaching task in which we could selectively manipulate task success and the variability of action outcomes, whilst also manipulating awareness of the dimension along which performance could be improved. Variability was manipulated either by amplifying natural errors, leaving the correlation between the executed movement and the visual feedback intact, or by adding extrinsic noise, decorrelating movement and feedback. We found that explicit, binary, feedback about success or failure was only sufficient for learning when participants were aware of the dimension along which motor behavior had to change. Without such awareness, learning was only present when extrinsic noise was added to the feedback, but not when task success or variability was manipulated in isolation; learning was also much slower. Our results highlight the importance of conscious awareness of the relevant dimension during motor learning, and suggest that higher-order moments of outcome signals are likely to play a significant role in skill learning in complex tasks.     

Correlations in state space can cause sub-optimal adaptation of optimal feedback control models

Control of our movements is apparently facilitated by an adaptive internal model in the cerebellum. It was long thought that this internal model implemented an adaptive inverse model and generated motor commands, but recently many reject that idea in favor of a forward model hypothesis. In theory, the forward model predicts upcoming state during reaching movements so the motor cortex can generate appropriate motor commands. Recent computational models of this process rely on the optimal feedback control (OFC) framework of control theory. OFC is a powerful tool for describing motor control, it does not describe adaptation. Some assume that adaptation of the forward model alone could explain motor adaptation, but this is widely understood to be overly simplistic. However, an adaptive optimal controller is difficult to implement. A reasonable alternative is to allow forward model adaptation to ‘re-tune’ the controller. Our simulations show that, as expected, forward model adaptation alone does not produce optimal trajectories during reaching movements perturbed by force fields. However, they also show that re-optimizing the controller from the forward model can be sub-optimal. This is because, in a system with state correlations or redundancies, accurate prediction requires different information than optimal control. We find that adding noise to the movements that matches noise found in human data is enough to overcome this problem. However, since the state space for control of real movements is far more complex than in our simple simulations, the effects of correlations on re-adaptation of the controller from the forward model cannot be overlooked.     

The cerebellum updates predictions about the visual consequences of one's behavior

Each action has sensory consequences that need to be distinguished from sensations arising from the environment. This is accomplished by the comparing of internal predictions about these consequences with the actual afference, thereby isolating the afferent component that is self-produced. Because the sensory consequences of actions vary as a result of changes of the effector's efficacy, internal predictions need to be updated continuously and on a short time scale. Here, we tested the hypothesis that this updating of predictions about the sensory consequences of actions is mediated by the cerebellum, a notion that parallels the cerebellum's role in motor learning. Patients with cerebellar lesions and their matched controls were equally able to detect experimental modifications of visual feedback about their pointing movements. When such feedback was constantly rotated, both groups instantly attributed the visual feedback to their own actions. However, in interleaved trials without actual feedback, patients did no longer account for this feedback rotation--neither perceptually nor with respect to motor performance. Both deficits can be explained by an impaired updating of internal predictions about the sensory consequences of actions caused by cerebellar pathology. Thus, the cerebellum guarantees both precise performance and veridical perceptual interpretation of actions.     

Normalized Index of Synergy for Evaluating the Coordination of Motor Commands

Humans perform various motor tasks by coordinating the redundant motor elements in their bodies. The coordination of motor outputs is produced by motor commands, as well properties of the musculoskeletal system. The aim of this study was to dissociate the coordination of motor commands from motor outputs. First, we conducted simulation experiments where the total elbow torque was generated by a model of a simple human right and left elbow with redundant muscles. The results demonstrated that muscle tension with signal-dependent noise formed a coordinated structure of trial-to-trial variability of muscle tension. Therefore, the removal of signal-dependent noise effects was required to evaluate the coordination of motor commands. We proposed a method to evaluate the coordination of motor commands, which removed signal-dependent noise from the measured variability of muscle tension. We used uncontrolled manifold analysis to calculate a normalized index of synergy. Simulation experiments confirmed that the proposed method could appropriately represent the coordinated structure of the variability of motor commands. We also conducted experiments in which subjects performed the same task as in the simulation experiments. The normalized index of synergy revealed that the subjects coordinated their motor commands to achieve the task. Finally, the normalized index of synergy was applied to a motor learning task to determine the utility of the proposed method. We hypothesized that a large part of the change in the coordination of motor outputs through learning was because of changes in motor commands. In a motor learning task, subjects tracked a target trajectory of the total torque. The change in the coordination of muscle tension through learning was dominated by that of motor commands, which supported the hypothesis. We conclude that the normalized index of synergy can be used to evaluate the coordination of motor commands independently from the properties of the musculoskeletal system.     

A neural signature of hierarchical reinforcement learning

Human behavior displays hierarchical structure: simple actions cohere into subtask sequences, which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive prediction arising from HRL. In ordinary reinforcement learning, reward prediction errors are computed when there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In three neuroimaging studies we observed neural responses consistent with such subgoal-related reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.     

Uncertainty–guided learning with scaled prediction errors in the basal ganglia

To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.