Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses

 Some characteristics of arm movements that humans exhibit during learning the dynamics of reaching are consistent with a theoretical framework where training results in motor commands that are gradually modified to predict and compensate for novel forces that may act on the hand. As a first approximation, the motor control system behaves as an adapting controller that learns an internal model of the dynamics of the task. It approximates inverse dynamics and predicts motor commands that are appropriate for a desired limb trajectory. However, we had previously noted that subtle motion characteristics observed during changes in task dynamics challenged this simple model and raised the possibility that adaptation also involved sensory–motor feedback pathways. These pathways reacted to sensory feedback during the course of the movement. Here we hypothesize that adaptation to dynamics might also involve a modification of how the CNS responds to sensory feedback. We tested this through experiments that quantified how the motor system's response to errors during voluntary movements changed as it adapted to dynamics of a force field. We describe a nonlinear approach that approximates the impedance of the arm, i.e., force response as a function of arm displacement trajectory. We observe that after adaptation, the impedance function changes in a way that closely matches and counters the effect of the force field. This is particularly prominent in the long-latency (>100 ms) component of response to perturbations. Therefore, it appears that practice not only modifies the internal model with which the brain generates motor commands that initiate a movement, but also the internal model with which sensory feedback is integrated with the ongoing descending commands in order to respond to error during the movement. Received: 10 January 2001 / Accepted in revised form: 30 May 2001     

Correlations in state space can cause sub-optimal adaptation of optimal feedback control models

Control of our movements is apparently facilitated by an adaptive internal model in the cerebellum. It was long thought that this internal model implemented an adaptive inverse model and generated motor commands, but recently many reject that idea in favor of a forward model hypothesis. In theory, the forward model predicts upcoming state during reaching movements so the motor cortex can generate appropriate motor commands. Recent computational models of this process rely on the optimal feedback control (OFC) framework of control theory. OFC is a powerful tool for describing motor control, it does not describe adaptation. Some assume that adaptation of the forward model alone could explain motor adaptation, but this is widely understood to be overly simplistic. However, an adaptive optimal controller is difficult to implement. A reasonable alternative is to allow forward model adaptation to ‘re-tune’ the controller. Our simulations show that, as expected, forward model adaptation alone does not produce optimal trajectories during reaching movements perturbed by force fields. However, they also show that re-optimizing the controller from the forward model can be sub-optimal. This is because, in a system with state correlations or redundancies, accurate prediction requires different information than optimal control. We find that adding noise to the movements that matches noise found in human data is enough to overcome this problem. However, since the state space for control of real movements is far more complex than in our simple simulations, the effects of correlations on re-adaptation of the controller from the forward model cannot be overlooked.     

A single-rate context-dependent learning process underlies rapid adaptation to familiar object dynamics

Motor learning has been extensively studied using dynamic (force-field) perturbations. These induce movement errors that result in adaptive changes to the motor commands. Several state-space models have been developed to explain how trial-by-trial errors drive the progressive adaptation observed in such studies. These models have been applied to adaptation involving novel dynamics, which typically occurs over tens to hundreds of trials, and which appears to be mediated by a dual-rate adaptation process. In contrast, when manipulating objects with familiar dynamics, subjects adapt rapidly within a few trials. Here, we apply state-space models to familiar dynamics, asking whether adaptation is mediated by a single-rate or dual-rate process. Previously, we reported a task in which subjects rotate an object with known dynamics. By presenting the object at different visual orientations, adaptation was shown to be context-specific, with limited generalization to novel orientations. Here we show that a multiple-context state-space model, with a generalization function tuned to visual object orientation, can reproduce the time-course of adaptation and de-adaptation as well as the observed context-dependent behavior. In contrast to the dual-rate process associated with novel dynamics, we show that a single-rate process mediates adaptation to familiar object dynamics. The model predicts that during exposure to the object across multiple orientations, there will be a degree of independence for adaptation and de-adaptation within each context, and that the states associated with all contexts will slowly de-adapt during exposure in one particular context. We confirm these predictions in two new experiments. Results of the current study thus highlight similarities and differences in the processes engaged during exposure to novel versus familiar dynamics. In both cases, adaptation is mediated by multiple context-specific representations. In the case of familiar object dynamics, however, the representations can be engaged based on visual context, and are updated by a single-rate process.     

An optimisation-based model for full-body upright reaching movements

An optimal simulation 3D model for full-body upright reaching movements was developed using graphic-based modelling tools (SimMechanics) to generate an inverse dynamics model of the skeleton and using parameterisation methods for a sensory motor controller. The adaptive weight coefficient of the cost function based on the final motor task error (i.e. distance between end-effector and target at the end of movement) was used to correct motor task error and physiological measurements (e.g. joint power, centre of mass displacement, etc.). The output of the simulation models using various cost functions were compared to experimental data from 15 healthy participants performing full-body upright reaching movements. The proposed method can reasonably predict full-body voluntary movements in terms of final posture, joint power, and movement of the centre of mass (COM) using simple algebraic calculations of inverse dynamics and forward kinematics instead of the complicated integrals of the forward dynamics. We found that the combination of several control strategies, i.e. minimising end-effector error, total joint power and body COM produced the best fit of the full-body reaching task.     

Modelling the control of interceptive actions

In recent years, several phenomenological dynamical models have been formulated that describe how perceptual variables are incorporated in the control of motor variables. We call these short-route models as they do not address how perception-action patterns might be constrained by the dynamical properties of the sensory, neural and musculoskeletal subsystems of the human action system. As an alternative, we advocate a long-route modelling approach in which the dynamics of these subsystems are explicitly addressed and integrated to reproduce interceptive actions. The approach is exemplified through a discussion of a recently developed model for interceptive actions consisting of a neural network architecture for the online generation of motor outflow commands, based on time-to-contact information and information about the relative positions and velocities of hand and ball. This network is shown to be consistent with both behavioural and neurophysiological data. Finally, some problems are discussed with regard to the question of how the motor outflow commands (i.e. the intended movement) might be modulated in view of the musculoskeletal dynamics.     

Conclusions on motor control depend on the type of model used to represent the periphery

Within the field of motor control, there is no consensus on which kinematic and kinetic aspects of movements are planned or controlled. Perturbing goal-directed movements is a frequently used tool to answer this question. To be able to draw conclusions about motor control from kinematic responses to perturbations, a model of the periphery (i.e., the skeleton, muscle–tendon complexes, and spinal reflex circuitry) is required. The purpose of the present study was to determine to what extent such conclusions depend on the level of simplification with which the dynamical properties of the periphery are modeled. For this purpose, we simulated fast goal-directed single-joint movement with four existing types of models. We tested how three types of perturbations affected movement trajectory if motor commands remained unchanged. We found that the four types of models of the periphery showed different robustness to the perturbations, leading to different predictions on how accurate motor commands need to be, i.e., how accurate the knowledge of external conditions needs to be. This means that when interpreting kinematic responses obtained in perturbation experiments the level of error correction attributed to adaptation of motor commands depends on the type of model used to describe the periphery.     

Inferior olive mirrors joint dynamics to implement an inverse controller

To produce smooth and coordinated motion, our nervous systems need to generate precisely timed muscle activation patterns that, due to axonal conduction delay, must be generated in a predictive and feedforward manner. Kawato proposed that the cerebellum accomplishes this by acting as an inverse controller that modulates descending motor commands to predictively drive the spinal cord such that the musculoskeletal dynamics are canceled out. This and other cerebellar theories do not, however, account for the rich biophysical properties expressed by the olivocerebellar complex’s various cell types, making these theories difficult to verify experimentally. Here we propose that a multizonal microcomplex’s (MZMC) inferior olivary neurons use their subthreshold oscillations to mirror a musculoskeletal joint’s underdamped dynamics, thereby achieving inverse control. We used control theory to map a joint’s inverse model onto an MZMC’s biophysics, and we used biophysical modeling to confirm that inferior olivary neurons can express the dynamics required to mirror biomechanical joints. We then combined both techniques to predict how experimentally injecting current into the inferior olive would affect overall motor output performance. We found that this experimental manipulation unmasked a joint’s natural dynamics, as observed by motor output ringing at the joint’s natural frequency, with amplitude proportional to the amount of current. These results support the proposal that the cerebellum—in particular an MZMC—is an inverse controller; the results also provide a biophysical implementation for this controller and allow one to make an experimentally testable prediction.     

Learning from sensory and reward prediction errors during motor adaptation

Voluntary motor commands produce two kinds of consequences. Initially, a sensory consequence is observed in terms of activity in our primary sensory organs (e.g., vision, proprioception). Subsequently, the brain evaluates the sensory feedback and produces a subjective measure of utility or usefulness of the motor commands (e.g., reward). As a result, comparisons between predicted and observed consequences of motor commands produce two forms of prediction error. How do these errors contribute to changes in motor commands? Here, we considered a reach adaptation protocol and found that when high quality sensory feedback was available, adaptation of motor commands was driven almost exclusively by sensory prediction errors. This form of learning had a distinct signature: as motor commands adapted, the subjects altered their predictions regarding sensory consequences of motor commands, and generalized this learning broadly to neighboring motor commands. In contrast, as the quality of the sensory feedback degraded, adaptation of motor commands became more dependent on reward prediction errors. Reward prediction errors produced comparable changes in the motor commands, but produced no change in the predicted sensory consequences of motor commands, and generalized only locally. Because we found that there was a within subject correlation between generalization patterns and sensory remapping, it is plausible that during adaptation an individual''s relative reliance on sensory vs. reward prediction errors could be inferred. We suggest that while motor commands change because of sensory and reward prediction errors, only sensory prediction errors produce a change in the neural system that predicts sensory consequences of motor commands.     

Prediction precedes control in motor learning

Skilled motor behavior relies on the brain learning both to control the body and predict the consequences of this control. Prediction turns motor commands into expected sensory consequences, whereas control turns desired consequences into motor commands. To capture this symmetry, the neural processes underlying prediction and control are termed the forward and inverse internal models, respectively. Here, we investigate how these two fundamental processes are related during motor learning. We used an object manipulation task in which subjects learned to move a hand-held object with novel dynamic properties along a prescribed path. We independently and simultaneously measured subjects' ability to control their actions and to predict their consequences. We found different time courses for predictor and controller learning, with prediction being learned far more rapidly than control. In early stages of manipulating the object, subjects could predict the consequences of their actions, as measured by the grip force they used to grasp the object, but could not generate appropriate actions for control, as measured by their hand trajectory. As predicted by several recent theoretical models of sensorimotor control, our results indicate that people can learn to predict the consequences of their actions before they can learn to control their actions.     

A mathematical model of the adaptive control of human arm motions

This paper discusses similarities between models of adaptive motor control suggested by recent experiments with human and animal subjects, and the structure of a new control law derived mathematically from nonlinear stability theory. In both models, the control actions required to track a specified trajectory are adaptively assembled from a large collection of simple computational elements. By adaptively recombining these elements, the controllers develop complex internal models which are used to compensate for the effects of externally imposed forces or changes in the physical properties of the system. On a motor learning task involving planar, multi-joint arm motions, the simulated performance of the mathematical model is shown to be qualitatively similar to observed human performance, suggesting that the model captures some of the interesting features of the dynamics of low-level motor adaptation. Received: 20 September 1994 / Accepted in revised form: 18 November 1998     

Implications of different classes of sensorimotor disturbance for cerebellar-based motor learning models

The exact role of the cerebellum in motor control and learning is not yet fully understood. The structure, connectivity and plasticity within cerebellar cortex has been extensively studied, but the patterns of connectivity and interaction with other brain structures, and the computational significance of these patterns, is less well known and a matter of debate. Two contrasting models of the role of the cerebellum in motor adaptation have previously been proposed. Most commonly, the cerebellum is employed in a purely feedforward pathway, with its output contributing directly to the outgoing motor command. The cerebellum must then learn an inverse model of the motor apparatus in order to achieve accurate control. More recently, Porrill et al. (Proc Biol Sci 271(1541):789–796, 2004) and Porrill et al. (PLoS Comput Biol 3:1935–1950, 2007a) and Porrill et al. (Neural Comput 19(1), 170–193, 2007b) have highlighted the potential importance of these recurrent connections by proposing an alternative architecture in which the cerebellum is embedded in a recurrent loop with brainstem control circuitry. In this framework, the feedforward connections are not necessary at all. The cerebellum must learn a forward model of the motor apparatus for accurate motor commands to be generated. We show here how these two models exhibit contrasting yet complimentary learning capabilities. Central to the differences in performance between architectures is that there are two distinct kinds of disturbance to which a motor system may need to adapt (1) changes in the relationship between the motor command and the observed outcome and (2) changes in the relationship between the stimulus and the desired outcome. The computational distinction between these two kinds of transformation is subtle and has therefore often been overlooked. However, the implications for learning turn out to be significant: learning with a feedforward architecture is robust following changes in the stimulus-desired outcome mapping but not necessarily the motor command-outcome mapping, while learning with a recurrent architecture is robust under changes in the motor command-outcome mapping but not necessarily the stimulus-desired outcome mapping. We first analyse these differences theoretically and through simulations in the vestibulo-ocular reflex (VOR), then illustrate how these same concepts apply more generally with a model of reaching movements.     

Adaptive control of movement deceleration during saccades

As you read this text, your eyes make saccades that guide your fovea from one word to the next. Accuracy of these movements require the brain to monitor and learn from visual errors. A current model suggests that learning is supported by two different adaptive processes, one fast (high error sensitivity, low retention), and the other slow (low error sensitivity, high retention). Here, we searched for signatures of these hypothesized processes and found that following experience of a visual error, there was an adaptive change in the motor commands of the subsequent saccade. Surprisingly, this adaptation was not uniformly expressed throughout the movement. Rather, after experience of a single error, the adaptive response in the subsequent trial was limited to the deceleration period. After repeated exposure to the same error, the acceleration period commands also adapted, and exhibited resistance to forgetting during set-breaks. In contrast, the deceleration period commands adapted more rapidly, but suffered from poor retention during these same breaks. State-space models suggested that acceleration and deceleration periods were supported by a shared adaptive state which re-aimed the saccade, as well as two separate processes which resembled a two-state model: one that learned slowly and contributed primarily via acceleration period commands, and another that learned rapidly but contributed primarily via deceleration period commands.     

Generalization in Adaptation to Stable and Unstable Dynamics

Humans skillfully manipulate objects and tools despite the inherent instability. In order to succeed at these tasks, the sensorimotor control system must build an internal representation of both the force and mechanical impedance. As it is not practical to either learn or store motor commands for every possible future action, the sensorimotor control system generalizes a control strategy for a range of movements based on learning performed over a set of movements. Here, we introduce a computational model for this learning and generalization, which specifies how to learn feedforward muscle activity in a function of the state space. Specifically, by incorporating co-activation as a function of error into the feedback command, we are able to derive an algorithm from a gradient descent minimization of motion error and effort, subject to maintaining a stability margin. This algorithm can be used to learn to coordinate any of a variety of motor primitives such as force fields, muscle synergies, physical models or artificial neural networks. This model for human learning and generalization is able to adapt to both stable and unstable dynamics, and provides a controller for generating efficient adaptive motor behavior in robots. Simulation results exhibit predictions consistent with all experiments on learning of novel dynamics requiring adaptation of force and impedance, and enable us to re-examine some of the previous interpretations of experiments on generalization.     

Internal models and intermittency: A theoretical account of human tracking behavior

This paper concerns the use of tracking studies to test a theoretical account of the information processing performed by the human CNS during control of movement. The theory provides a bridge between studies of reaction time and continuous tracking. It is proposed that the human CNS includes neuronal circuitry to compute inverse internal models of the multiple input, multiple output, dynamic, nonlinear relationships between outgoing motor commands and their resulting perceptual consequences. The inverse internal models are employed during movement execution to transform preplanned trajectories of desired perceptual consequences into appropriate outgoing motor commands to achieve them. A finite interval of time is required by the CNS to preplan the desired perceptual consequences of a movement and it does not commence planning a new movement until planning of the old one has been completed. This behavior introduces intermittency into the planning of movements. In this paper we show that the gain and phase frequency response characteristics of the human operator in a visual pursuit tracking task can be derived theoretically from these assumptions. By incorporating the effects of internal model inaccuracy and of speed-accuracy trade-off in performance, it is shown that various aspects of experimentally measured tracking behavior can be accounted for.     

Adaptive dynamic programming as a theory of sensorimotor control

Many characteristics of sensorimotor control can be explained by models based on optimization and optimal control theories. However, most of the previous models assume that the central nervous system has access to the precise knowledge of the sensorimotor system and its interacting environment. This viewpoint is difficult to be justified theoretically and has not been convincingly validated by experiments. To address this problem, this paper presents a new computational mechanism for sensorimotor control from a perspective of adaptive dynamic programming (ADP), which shares some features of reinforcement learning. The ADP-based model for sensorimotor control suggests that a command signal for the human movement is derived directly from the real-time sensory data, without the need to identify the system dynamics. An iterative learning scheme based on the proposed ADP theory is developed, along with rigorous convergence analysis. Interestingly, the computational model as advocated here is able to reproduce the motor learning behavior observed in experiments where a divergent force field or velocity-dependent force field was present. In addition, this modeling strategy provides a clear way to perform stability analysis of the overall system. Hence, we conjecture that human sensorimotor systems use an ADP-type mechanism to control movements and to achieve successful adaptation to uncertainties present in the environment.     

Internal models for motor control and trajectory planning

A number of internal model concepts are now widespread in neuroscience and cognitive science. These concepts are supported by behavioral, neurophysiological, and imaging data; furthermore, these models have had their structures and functions revealed by such data. In particular, a specific theory on inverse dynamics model learning is directly supported by unit recordings from cerebellar Purkinje cells. Multiple paired forward inverse models describing how diverse objects and environments can be controlled and learned separately have recently been proposed. The 'minimum variance model' is another major recent advance in the computational theory of motor control. This model integrates two furiously disputed approaches on trajectory planning, strongly suggesting that both kinematic and dynamic internal models are utilized in movement planning and control.     

An internal model architecture for novelty detection: implications for cerebellar and collicular roles in sensory processing

The cerebellum is thought to implement internal models for sensory prediction, but details of the underlying circuitry are currently obscure. We therefore investigated a specific example of internal-model based sensory prediction, namely detection of whisker contacts during whisking. Inputs from the vibrissae in rats can be affected by signals generated by whisker movement, a phenomenon also observable in whisking robots. Robot novelty-detection can be improved by adaptive noise-cancellation, in which an adaptive filter learns a forward model of the whisker plant that allows the sensory effects of whisking to be predicted and thus subtracted from the noisy sensory input. However, the forward model only uses information from an efference copy of the whisking commands. Here we show that the addition of sensory information from the whiskers allows the adaptive filter to learn a more complex internal model that performs more robustly than the forward model, particularly when the whisking-induced interference has a periodic structure. We then propose a neural equivalent of the circuitry required for adaptive novelty-detection in the robot, in which the role of the adaptive filter is carried out by the cerebellum, with the comparison of its output (an estimate of the self-induced interference) and the original vibrissal signal occurring in the superior colliculus, a structure noted for its central role in novelty detection. This proposal makes a specific prediction concerning the whisker-related functions of a region in cerebellar cortical zone A(2) that in rats receives climbing fibre input from the superior colliculus (via the inferior olive). This region has not been observed in non-whisking animals such as cats and primates, and its functional role in vibrissal processing has hitherto remained mysterious. Further investigation of this system may throw light on how cerebellar-based internal models could be used in broader sensory, motor and cognitive contexts.     

Analysis of kinematic invariances of multijoint reaching movement

 There is a no unique relationship between the trajectory of the hand, represented in cartesian or extrinsic space, and its trajectory in joint angle or intrinsic space in the general condition of joint redundancy. The goal of this work is to analyze the relation between planning the trajectory of a multijoint movement in these two coordinate systems. We show that the cartesian trajectory can be planned based on the task parameters (target coordinates, etc.) prior to and independently of angular trajectories. Angular time profiles are calculated from the cartesian trajectory to serve as a basis for muscle control commands. A unified differential equation that allows planning trajectories in cartesian and angular spaces simultaneously is proposed. Due to joint redundancy, each cartesian trajectory corresponds to a family of angular trajectories which can account for the substantial variability of the latter. A set of strategies for multijoint motor control following from this model is considered; one of them coincides with the frog wiping reflex model and resolves the kinematic inverse problem without inversion. The model trajectories exhibit certain properties observed in human multijoint reaching movements such as movement equifinality, straight end-point paths, bell-shaped tangential velocity profiles, speed-sensitive and speed-insensitive movement strategies, peculiarities of the response to double-step targets, and variations of angular trajectory without variations of the limb end-point trajectory in cartesian space. In humans, those properties are almost independent of limb configuration, target location, movement duration, and load. In the model, these properties are invariant to an affine transform of cartesian space. This implies that these properties are not a special goal of the motor control system but emerge from movement kinematics that reflect limb geometry, dynamics, and elementary principles of motor control used in planning. All the results are given analytically and, in order to compare the model with experimental results, by computer simulations. Received: 6 April 1994/Accepted in revised form: 25 April 1995     

Motor learning through the combination of primitives

In this paper we discuss a new perspective on how the central nervous system (CNS) represents and solves some of the most fundamental computational problems of motor control. In particular, we consider the task of transforming a planned limb movement into an adequate set of motor commands. To carry out this task the CNS must solve a complex inverse dynamic problem. This problem involves the transformation from a desired motion to the forces that are needed to drive the limb. The inverse dynamic problem is a hard computational challenge because of the need to coordinate multiple limb segments and because of the continuous changes in the mechanical properties of the limbs and of the environment with which they come in contact. A number of studies of motor learning have provided support for the idea that the CNS creates, updates and exploits internal representations of limb dynamics in order to deal with the complexity of inverse dynamics. Here we discuss how such internal representations are likely to be built by combining the modular primitives in the spinal cord as well as other building blocks found in higher brain structures. Experimental studies on spinalized frogs and rats have led to the conclusion that the premotor circuits within the spinal cord are organized into a set of discrete modules. Each module, when activated, induces a specific force field and the simultaneous activation of multiple modules leads to the vectorial combination of the corresponding fields. We regard these force fields as computational primitives that are used by the CNS for generating a rich grammar of motor behaviours.     

Cerebellar learning of accurate predictive control for fast-reaching movements

Long conduction delays in the nervous system prevent the accurate control of movements by feedback control alone. We present a new, biologically plausible cerebellar model to study how fast arm movements can be executed in spite of these delays. To provide a realistic test-bed of the cerebellar neural model, we embed the cerebellar network in a simulated biological motor system comprising a spinal cord model and a six-muscle two-dimensional arm model. We argue that if the trajectory errors are detected at the spinal cord level, memory traces in the cerebellum can solve the temporal mismatch problem between efferent motor commands and delayed error signals. Moreover, learning is made stable by the inclusion of the cerebello-nucleo-olivary loop in the model. It is shown that the cerebellar network implements a nonlinear predictive regulator by learning part of the inverse dynamics of the plant and spinal circuit. After learning, fast accurate reaching movements can be generated. Received: 8 February 1999 /Accepted in revised form: 7 August 1999