Learning with slight forgetting optimizes sensorimotor transformation in redundant motor systems

Recent theoretical studies have proposed that the redundant motor system in humans achieves well-organized stereotypical movements by minimizing motor effort cost and motor error. However, it is unclear how this optimization process is implemented in the brain, presumably because conventional schemes have assumed a priori that the brain somehow constructs the optimal motor command, and largely ignored the underlying trial-by-trial learning process. In contrast, recent studies focusing on the trial-by-trial modification of motor commands based on error information suggested that forgetting (i.e., memory decay), which is usually considered as an inconvenient factor in motor learning, plays an important role in minimizing the motor effort cost. Here, we examine whether trial-by-trial error-feedback learning with slight forgetting could minimize the motor effort and error in a highly redundant neural network for sensorimotor transformation and whether it could predict the stereotypical activation patterns observed in primary motor cortex (M1) neurons. First, using a simple linear neural network model, we theoretically demonstrated that: 1) this algorithm consistently leads the neural network to converge at a unique optimal state; 2) the biomechanical properties of the musculoskeletal system necessarily determine the distribution of the preferred directions (PD; the direction in which the neuron is maximally active) of M1 neurons; and 3) the bias of the PDs is steadily formed during the minimization of the motor effort. Furthermore, using a non-linear network model with realistic musculoskeletal data, we demonstrated numerically that this algorithm could consistently reproduce the PD distribution observed in various motor tasks, including two-dimensional isometric torque production, two-dimensional reaching, and even three-dimensional reaching tasks. These results may suggest that slight forgetting in the sensorimotor transformation network is responsible for solving the redundancy problem in motor control.     

Correlations in state space can cause sub-optimal adaptation of optimal feedback control models

Control of our movements is apparently facilitated by an adaptive internal model in the cerebellum. It was long thought that this internal model implemented an adaptive inverse model and generated motor commands, but recently many reject that idea in favor of a forward model hypothesis. In theory, the forward model predicts upcoming state during reaching movements so the motor cortex can generate appropriate motor commands. Recent computational models of this process rely on the optimal feedback control (OFC) framework of control theory. OFC is a powerful tool for describing motor control, it does not describe adaptation. Some assume that adaptation of the forward model alone could explain motor adaptation, but this is widely understood to be overly simplistic. However, an adaptive optimal controller is difficult to implement. A reasonable alternative is to allow forward model adaptation to ‘re-tune’ the controller. Our simulations show that, as expected, forward model adaptation alone does not produce optimal trajectories during reaching movements perturbed by force fields. However, they also show that re-optimizing the controller from the forward model can be sub-optimal. This is because, in a system with state correlations or redundancies, accurate prediction requires different information than optimal control. We find that adding noise to the movements that matches noise found in human data is enough to overcome this problem. However, since the state space for control of real movements is far more complex than in our simple simulations, the effects of correlations on re-adaptation of the controller from the forward model cannot be overlooked.     

Model-based development of neuroprosthesis for paraplegic patients.

In paraplegic patients with upper motor neuron lesions the signal path from the central nervous system to the muscles is interrupted. Functional electrical stimulation applied to the lower motor neurons can replace the lacking signals. A so-called neuroprosthesis may be used to restore motor function in paraplegic patients on the basis of functional electrical stimulation. However, the control of multiple joints is difficult due to the complexity, nonlinearity, and time-variance of the system involved. Furthermore, effects such as muscle fatigue, spasticity, and limited force in the stimulated muscle further complicate the control task. Mathematical models of the human musculoskeletal system can support the development of neuroprosthesis. In this article a detailed overview of the existing work in the literature is given and two examples developed by the author are presented that give an insight into model-based development of neuroprosthesis for paraplegic patients. It is shown that modelling the musculoskeletal system can provide better understanding of muscular force production and movement coordination principles. Models can also be used to design and test stimulation patterns and feedback control strategies. Additionally, model components can be implemented in a controller to improve control performance. Eventually, the use of musculoskeletal models for neuroprosthesis design may help to avoid internal disturbances such as fatigue and optimize muscular force output. Furthermore, better controller quality can be obtained than in previous empirical approaches. In addition, the number of experimental tests to be performed with human subjects can be reduced. It is concluded that mathematical models play an increasing role in the development of reliable closed-loop controlled, lower extremity neuroprostheses.     

Active stiffness of the ankle in response to inertial and elastic loads.

Effective stiffness of the musculoskeletal system was examined as a function of the characteristics of an external load. Thirteen healthy subjects provided active contraction of the ankle plantarflexion musculature in a neutral ankle posture to support an external load. Musculoskeletal stiffness was computed from kinetic data recorded in response to dorsiflexion/plantarflexion perturbations. Ankle dynamics were recorded while supporting external loads of 19 and 38 kg with and without antagonistic co-contraction. External loads were applied using pure gravitational mass. In separate trials external loads were applied from stretch of steel springs in parallel with the plantarflexion musculature that also provided added parallel stiffness to the system. Adding external stiffness of 4.9 and 8.1 kN/m surprisingly failed to significantly change the stiffness of the ankle-plus-spring system. This suggests contributions from intrinsic muscle stiffness and reflex stiffness declined in response to added external stiffness. This could not be explained by load magnitudes, ankle postures, or co-activation as these were similar between the inertial and elastic loading conditions. However, non-linear parametric analyses suggest mean intrinsic stiffness of 35.5 kN/m and reflex gain of 11.6 kN/m with a constant reflex delay of 70 ms accurately described the empirical results. The phase response between the mechanical dynamics of the musculoskeletal system and delayed neuromotor feedback combine to provide robust control of system behavior.     

Non-linear stimulus-response behavior of the human stance control system is predicted by optimization of a system with sensory and motor noise

We developed a theory of human stance control that predicted (1) how subjects re-weight their utilization of proprioceptive and graviceptive orientation information in experiments where eyes closed stance was perturbed by surface-tilt stimuli with different amplitudes, (2) the experimentally observed increase in body sway variability (i.e. the “remnant” body sway that could not be attributed to the stimulus) with increasing surface-tilt amplitude, (3) neural controller feedback gains that determine the amount of corrective torque generated in relation to sensory cues signaling body orientation, and (4) the magnitude and structure of spontaneous body sway. Responses to surface-tilt perturbations with different amplitudes were interpreted using a feedback control model to determine control parameters and changes in these parameters with stimulus amplitude. Different combinations of internal sensory and/or motor noise sources were added to the model to identify the properties of noise sources that were able to account for the experimental remnant sway characteristics. Various behavioral criteria were investigated to determine if optimization of these criteria could predict the identified model parameters and amplitude-dependent parameter changes. Robust findings were that remnant sway characteristics were best predicted by models that included both sensory and motor noise, the graviceptive noise magnitude was about ten times larger than the proprioceptive noise, and noise sources with signal-dependent properties provided better explanations of remnant sway. Overall results indicate that humans dynamically weight sensory system contributions to stance control and tune their corrective responses to minimize the energetic effects of sensory noise and external stimuli.     

Noise Normalizes Firing Output of Mouse Lateral Geniculate Nucleus Neurons

The output of individual neurons is dependent on both synaptic and intrinsic membrane properties. While it is clear that the response of an individual neuron can be facilitated or inhibited based on the summation of its constituent synaptic inputs, it is not clear whether subthreshold activity, (e.g. synaptic “noise”- fluctuations that do not change the mean membrane potential) also serve a function in the control of neuronal output. Here we studied this by making whole-cell patch-clamp recordings from 29 mouse thalamocortical relay (TC) neurons. For each neuron we measured neuronal gain in response to either injection of current noise, or activation of the metabotropic glutamate receptor-mediated cortical feedback network (synaptic noise). As expected, injection of current noise via the recording pipette induces shifts in neuronal gain that are dependent on the amplitude of current noise, such that larger shifts in gain are observed in response to larger amplitude noise injections. Importantly we show that shifts in neuronal gain are also dependent on the intrinsic sensitivity of the neuron tested, such that the gain of intrinsically sensitive neurons is attenuated divisively in response to current noise, while the gain of insensitive neurons is facilitated multiplicatively by injection of current noise- effectively normalizing the output of the dLGN as a whole. In contrast, when the cortical feedback network was activated, only multiplicative gain changes were observed. These network activation-dependent changes were associated with reductions in the slow afterhyperpolarization (sAHP), and were mediated at least in part, by T-type calcium channels. Together, this suggests that TC neurons have the machinery necessary to compute multiple output solutions to a given set of stimuli depending on the current level of network stimulation.     

Analysis of reflex modulation with a biologically realistic neural network

In this study, a neuromusculoskeletal model was built to give insight into the mechanisms behind the modulation of reflexive feedback strength as experimentally identified in the human shoulder joint. The model is an integration of a biologically realistic neural network consisting of motoneurons and interneurons, modeling 12 populations of spinal neurons, and a one degree-of-freedom musculoskeletal model, including proprioceptors. The model could mimic the findings of human postural experiments, using presynaptic inhibition of the Ia afferents to modulate the feedback gains. In a pathological case, disabling one specific neural connection between the inhibitory interneurons and the motoneurons could mimic the experimental findings in complex regional pain syndrome patients. It is concluded that the model is a valuable tool to gain insight into the spinal contributions to human motor control. Applications lay in the fields of human motor control and neurological disorders, where hypotheses on motor dysfunction can be tested, like spasticity, clonus, and tremor. Action Editor: Karen Sigvardt     

Noise effects on robust synchronization of a small pacemaker neuronal ensemble via nonlinear controller: electronic circuit design

In this paper, we report on the synchronization of a pacemaker neuronal ensemble constituted of an AB neuron electrically coupled to two PD neurons. By the virtue of this electrical coupling, they can fire synchronous bursts of action potential. An external master neuron is used to induce to the whole system the desired dynamics, via a nonlinear controller. Such controller is obtained by a combination of sliding mode and feedback control. The proposed controller is able to offset uncertainties in the synchronized systems. We show how noise affects the synchronization of the pacemaker neuronal ensemble, and briefly discuss its potential benefits in our synchronization scheme. An extended Hindmarsh–Rose neuronal model is used to represent a single cell dynamic of the network. Numerical simulations and Pspice implementation of the synchronization scheme are presented. We found that, the proposed controller reduces the stochastic resonance of the network when its gain increases.     

Normalized Index of Synergy for Evaluating the Coordination of Motor Commands

Humans perform various motor tasks by coordinating the redundant motor elements in their bodies. The coordination of motor outputs is produced by motor commands, as well properties of the musculoskeletal system. The aim of this study was to dissociate the coordination of motor commands from motor outputs. First, we conducted simulation experiments where the total elbow torque was generated by a model of a simple human right and left elbow with redundant muscles. The results demonstrated that muscle tension with signal-dependent noise formed a coordinated structure of trial-to-trial variability of muscle tension. Therefore, the removal of signal-dependent noise effects was required to evaluate the coordination of motor commands. We proposed a method to evaluate the coordination of motor commands, which removed signal-dependent noise from the measured variability of muscle tension. We used uncontrolled manifold analysis to calculate a normalized index of synergy. Simulation experiments confirmed that the proposed method could appropriately represent the coordinated structure of the variability of motor commands. We also conducted experiments in which subjects performed the same task as in the simulation experiments. The normalized index of synergy revealed that the subjects coordinated their motor commands to achieve the task. Finally, the normalized index of synergy was applied to a motor learning task to determine the utility of the proposed method. We hypothesized that a large part of the change in the coordination of motor outputs through learning was because of changes in motor commands. In a motor learning task, subjects tracked a target trajectory of the total torque. The change in the coordination of muscle tension through learning was dominated by that of motor commands, which supported the hypothesis. We conclude that the normalized index of synergy can be used to evaluate the coordination of motor commands independently from the properties of the musculoskeletal system.     

Neural Coding of Finger and Wrist Movements

Previous work (Schieber and Hibbard, 1993) has shown that single motor cortical neurons do not discharge specifically for a particular flexion-extension finger movement but instead are active with movements of different fingers. In addition, neuronal populations active with movements of different fingers overlap extensively in their spatial locations in the motor cortex. These data suggested that control of any finger movement utilizes a distributed population of neurons. In this study we applied the neuronal population vector analysis (Georgopoulos et al., 1983) to these same data to determine (1) whether single cells are tuned in an abstract, three-dimensional (3D) instructed finger and wrist movement space with hand-like geometry and (2) whether the neuronal population encodes specific finger movements. We found that the activity of 132/176 (75%) motor cortical neurons related to finger movements was indeed tuned in this space. Moreover, the population vector computed in this space predicted well the instructed finger movement. Thus, although single neurons may be related to several disparate finger movements, and neurons related to different finger movements are intermingled throughout the hand area of the motor cortex, the neuronal population activity does specify particular finger movements.     

Learning combined feedback and feedforward control of a musculoskeletal system

The goal of this paper is the learning of neuromuscular control, given the following necessary conditions: (1) time delays in the control loop, (2) non-linear muscle characteristics, (3) learning of feedforward and feedback control, (4) possibility of feedback gain modulation during a task. A control system and learning methodology that satisfy those conditions is given. The control system contains a neural network, comprising both feedforward and feedback control. The learning method is backpropagation through time with an explicit sensitivity model. Results will be given for a one degree of freedom arm with two muscles. Good control results are achieved which compare well with experimental data. Analysis of the controller shows that significant differences in controller characteristics are found if the loop delays are neglected. During a control task the system shows feedback gain modulation, similar to experimentally found reflex gain modulation during rapid voluntary contraction. If only limited feedback information is available to the controller the system learns to co-contract the antagonistic muscle pair. In this way joint stiffness increases and stable control is more easily maintained. Received: 7 November 1995 / Accepted in revised form: 13 February 1996     

An approximate stochastic optimal control framework to simulate nonlinear neuro-musculoskeletal models in the presence of noise

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.     

Characteristics of synaptic connections between rodent primary somatosensory and motor cortices

The reciprocal connections between primary motor (M1) and primary somatosensory cortices (S1) are hypothesized to play a crucial role in the ability to update motor plans in response to changes in the sensory periphery. These interactions provide M1 with information about the sensory environment that in turn signals S1 with anticipatory knowledge of ongoing motor plans. In order to examine the synaptic basis of sensorimotor feedforward (S1–M1) and feedback (M1–S1) connections directly, we utilized whole-cell recordings in slices that preserve these reciprocal sensorimotor connections. Our findings indicate that these regions are connected via direct monosynaptic connections in both directions. Larger magnitude responses were observed in the feedforward direction (S1–M1), while the feedback (M1–S1) responses occurred at shorter latencies. The morphology as well as the intrinsic firing properties of the neurons in these pathways indicates that both excitatory and inhibitory neurons are targeted. Differences in synaptic physiology suggest that there exist specializations within the sensorimotor pathway that may allow for the rapid updating of sensory–motor processing within the cortex in response to changes in the sensory periphery.     

Characteristics of synaptic connections between rodent primary somatosensory and motor cortices

The reciprocal connections between primary motor (M1) and primary somatosensory cortices (S1) are hypothesized to play a crucial role in the ability to update motor plans in response to changes in the sensory periphery. These interactions provide M1 with information about the sensory environment that in turn signals S1 with anticipatory knowledge of ongoing motor plans. In order to examine the synaptic basis of sensorimotor feedforward (S1-M1) and feedback (M1-S1) connections directly, we utilized whole-cell recordings in slices that preserve these reciprocal sensorimotor connections. Our findings indicate that these regions are connected via direct monosynaptic connections in both directions. Larger magnitude responses were observed in the feedforward direction (S1-M1), while the feedback (M1-S1) responses occurred at shorter latencies. The morphology as well as the intrinsic firing properties of the neurons in these pathways indicates that both excitatory and inhibitory neurons are targeted. Differences in synaptic physiology suggest that there exist specializations within the sensorimotor pathway that may allow for the rapid updating of sensory-motor processing within the cortex in response to changes in the sensory periphery.     

A model of neuro-musculo-skeletal system for human locomotion under position constraint condition

The human locomotion was studied on the basis of the interaction of the musculo-skeletal system, the neural system and the environment. A mathematical model of human locomotion under position constraint condition was established. Besides the neural rhythm generator, the posture controller and the sensory system, the environment feedback controller and the stability controller were taken into account in the model. The environment feedback controller was proposed for two purposes, obstacle avoidance and target position control of the swing foot. The stability controller was proposed to imitate the self-balancing ability of a human body and improve the stability of the model. In the stability controller, the ankle torque was used to control the velocity of the body gravity center. A prediction control algorithm was applied to calculate the torque magnitude of the stability controller. As an example, human stairs climbing movement was simulated and the results were given. The simulation result proved that the mathematical modeling of the task was successful.     

Inferior olive mirrors joint dynamics to implement an inverse controller

To produce smooth and coordinated motion, our nervous systems need to generate precisely timed muscle activation patterns that, due to axonal conduction delay, must be generated in a predictive and feedforward manner. Kawato proposed that the cerebellum accomplishes this by acting as an inverse controller that modulates descending motor commands to predictively drive the spinal cord such that the musculoskeletal dynamics are canceled out. This and other cerebellar theories do not, however, account for the rich biophysical properties expressed by the olivocerebellar complex’s various cell types, making these theories difficult to verify experimentally. Here we propose that a multizonal microcomplex’s (MZMC) inferior olivary neurons use their subthreshold oscillations to mirror a musculoskeletal joint’s underdamped dynamics, thereby achieving inverse control. We used control theory to map a joint’s inverse model onto an MZMC’s biophysics, and we used biophysical modeling to confirm that inferior olivary neurons can express the dynamics required to mirror biomechanical joints. We then combined both techniques to predict how experimentally injecting current into the inferior olive would affect overall motor output performance. We found that this experimental manipulation unmasked a joint’s natural dynamics, as observed by motor output ringing at the joint’s natural frequency, with amplitude proportional to the amount of current. These results support the proposal that the cerebellum—in particular an MZMC—is an inverse controller; the results also provide a biophysical implementation for this controller and allow one to make an experimentally testable prediction.     

Recovery in stroke rehabilitation through the rotation of preferred directions induced by bimanual movements: a computational study

Stroke patients recover more effectively when they are rehabilitated with bimanual movement rather than with unimanual movement; however, it remains unclear why bimanual movement is more effective for stroke recovery. Using a computational model of stroke recovery, this study suggests that bimanual movement facilitates the reorganization of a damaged motor cortex because this movement induces rotations in the preferred directions (PDs) of motor cortex neurons. Although the tuning curves of these neurons differ during unimanual and bimanual movement, changes in PD, but not changes in modulation depth, facilitate such reorganization. In addition, this reorganization was facilitated only when encoding PDs are rotated, but decoding PDs are not rotated. Bimanual movement facilitates reorganization because this movement changes neural activities through inter-hemispheric inhibition without changing cortical-spinal-muscle connections. Furthermore, stronger inter-hemispheric inhibition between motor cortices results in more effective reorganization. Thus, this study suggests that bimanual movement is effective for stroke rehabilitation because this movement rotates the encoding PDs of motor cortex neurons.     

Feedback-induced gain control in stochastic spiking networks

The joint influence of recurrent feedback and noise on gain control in a network of globally coupled spiking leaky integrate-and-fire neurons is studied theoretically and numerically. The context of our work is the origin of divisive versus subtractive gain control, as mixtures of these effects are seen in a variety of experimental systems. We focus on changes in the slope of the mean firing frequency-versus-input bias (f –I) curve when the gain control signal to the cells comes from the cells’ output spikes. Feedback spikes are modeled as alpha functions that produce an additive current in the current balance equation. For generality, they occur after a fixed minimum delay. We show that purely divisive gain control, i.e. changes in the slope of the f –I curve, arises naturally with this additive negative or positive feedback, due to a linearizing actions of feedback. Negative feedback alone lowers the gain, accounting in particular for gain changes in weakly electric fish upon pharmacological opening of the feedback loop as reported by Bastian (J Neurosci 6:553–562, 1986). When negative feedback is sufficiently strong it further causes oscillatory firing patterns which produce irregularities in the f –I curve. Small positive feedback alone increases the gain, but larger amounts cause abrupt jumps to higher firing frequencies. On the other hand, noise alone in open loop linearizes the f –I curve around threshold, and produces mixtures of divisive and subtractive gain control. With both noise and feedback, the combined gain control schemes produce a primarily divisive gain control shift, indicating the robustness of feedback gain control in stochastic networks. Similar results are found when the “input” parameter is the contrast of a time-varying signal rather than the bias current. Theoretical results are derived relating the slope of the f –I curve to feedback gain and noise strength. Good agreement with simulation results are found for inhibitory and excitatory feedback. Finally, divisive feedback is also found for conductance-based feedback (shunting or excitatory) with and without noise. This article is part of a special issue on Neuronal Dynamics of Sensory Coding.     

Modeling Inter-trial Variability of Saccade Trajectories: Effects of Lesions of the Oculomotor Part of the Fastigial Nucleus

This study investigates the inter-trial variability of saccade trajectories observed in five rhesus macaques (Macaca mulatta). For each time point during a saccade, the inter-trial variance of eye position and its covariance with eye end position were evaluated. Data were modeled by a superposition of three noise components due to 1) planning noise, 2) signal-dependent motor noise, and 3) signal-dependent premotor noise entering within an internal feedback loop. Both planning noise and signal-dependent motor noise (together called accumulating noise) predict a simple S-shaped variance increase during saccades, which was not sufficient to explain the data. Adding noise within an internal feedback loop enabled the model to mimic variance/covariance structure in each monkey, and to estimate the noise amplitudes and the feedback gain. Feedback noise had little effect on end point noise, which was dominated by accumulating noise. This analysis was further extended to saccades executed during inactivation of the caudal fastigial nucleus (cFN) on one side of the cerebellum. Saccades ipsiversive to an inactivated cFN showed more end point variance than did normal saccades. During cFN inactivation, eye position during saccades was statistically more strongly coupled to eye position at saccade end. The proposed model could fit the variance/covariance structure of ipsiversive and contraversive saccades. Inactivation effects on saccade noise are explained by a decrease of the feedback gain and an increase of planning and/or signal-dependent motor noise. The decrease of the fitted feedback gain is consistent with previous studies suggesting a role for the cerebellum in an internal feedback mechanism. Increased end point variance did not result from impaired feedback but from the increase of accumulating noise. The effects of cFN inactivation on saccade noise indicate that the effects of cFN inactivation cannot be explained entirely with the cFN’s direct connections to the saccade-related premotor centers in the brainstem.     

Assessing the function of motor cortex: single-neuron models of how neural response is modulated by limb biomechanics

Do neurons in primary motor cortex encode the generative details of motor behavior, such as individual muscle activities, or do they encode high-level movement attributes? Resolving this question has proven difficult, in large part because of the sizeable uncertainty inherent in estimating or measuring the joint torques and muscle forces that underlie movements made by biological limbs. We circumvented this difficulty by considering single-neuron responses in an isometric task, where joint torques and muscle forces can be straightforwardly computed from limb geometry. The response for each neuron was modeled as a linear function of a "preferred" joint torque vector, and this model was fit to individual neural responses across variations in limb posture. The resulting goodness of fit suggests that neurons in motor cortex do encode the kinetics of motor behavior and that the neural response properties of "preferred direction" and "gain" are dual components of a unitary response vector.