A mathematical modeling study of inter-segmental coordination during stick insect walking

The biomechanical conditions for walking in the stick insect require a modeling approach that is based on the control of pairs of antagonistic motoneuron (MN) pools for each leg joint by independent central pattern generators (CPGs). Each CPG controls a pair of antagonistic MN pools. Furthermore, specific sensory feedback signals play an important role in the control of single leg movement and in the generation of inter-leg coordination or the interplay between both tasks. Currently, however, no mathematical model exists that provides a theoretical approach to understanding the generation of coordinated locomotion in such a multi-legged locomotor system. In the present study, I created such a theoretical model for the stick insect walking system, which describes the MN activity of a single forward stepping middle leg and helps to explain the neuronal mechanisms underlying coordinating information transfer between ipsilateral legs. In this model, CPGs that belong to the same leg, as well as those belonging to different legs, are connected by specific sensory feedback pathways that convey information about movements and forces generated during locomotion. The model emphasizes the importance of sensory feedback, which is used by the central nervous system to enhance weak excitatory and inhibitory synaptic connections from front to rear between the three thorax-coxa-joint CPGs. Thereby the sensory feedback activates caudal pattern generation networks and helps to coordinate leg movements by generating in-phase and out-of-phase thoracic MN activity.     

Shaping embodied neural networks for adaptive goal-directed behavior

The acts of learning and memory are thought to emerge from the modifications of synaptic connections between neurons, as guided by sensory feedback during behavior. However, much is unknown about how such synaptic processes can sculpt and are sculpted by neuronal population dynamics and an interaction with the environment. Here, we embodied a simulated network, inspired by dissociated cortical neuronal cultures, with an artificial animal (an animat) through a sensory-motor loop consisting of structured stimuli, detailed activity metrics incorporating spatial information, and an adaptive training algorithm that takes advantage of spike timing dependent plasticity. By using our design, we demonstrated that the network was capable of learning associations between multiple sensory inputs and motor outputs, and the animat was able to adapt to a new sensory mapping to restore its goal behavior: move toward and stay within a user-defined area. We further showed that successful learning required proper selections of stimuli to encode sensory inputs and a variety of training stimuli with adaptive selection contingent on the animat's behavior. We also found that an individual network had the flexibility to achieve different multi-task goals, and the same goal behavior could be exhibited with different sets of network synaptic strengths. While lacking the characteristic layered structure of in vivo cortical tissue, the biologically inspired simulated networks could tune their activity in behaviorally relevant manners, demonstrating that leaky integrate-and-fire neural networks have an innate ability to process information. This closed-loop hybrid system is a useful tool to study the network properties intermediating synaptic plasticity and behavioral adaptation. The training algorithm provides a stepping stone towards designing future control systems, whether with artificial neural networks or biological animats themselves.     

Learning to control a brain-machine interface for reaching and grasping by primates

Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain–machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.     

Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates

Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain–machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.     

Comparison of human and humanoid robot control of upright stance

There is considerable recent interest in developing humanoid robots. An important substrate for many motor actions in both humans and biped robots is the ability to maintain a statically or dynamically stable posture. Given the success of the human design, one would expect there are lessons to be learned in formulating a postural control mechanism for robots. In this study we limit ourselves to considering the problem of maintaining upright stance. Human stance control is compared to a suggested method for robot stance control called zero moment point (ZMP) compensation. Results from experimental and modeling studies suggest there are two important subsystems that account for the low- and mid-frequency (DC to 1 Hz) dynamic characteristics of human stance control. These subsystems are (1) a “sensory integration” mechanism whereby orientation information from multiple sensory systems encoding body kinematics (i.e. position, velocity) is flexibly combined to provide an overall estimate of body orientation while allowing adjustments (sensory re-weighting) that compensate for changing environmental conditions and (2) an “effort control” mechanism that uses kinetic-related (i.e., force-related) sensory information to reduce the mean deviation of body orientation from upright. Functionally, ZMP compensation is directly analogous to how humans appear to use kinetic feedback to modify the main sensory integration feedback loop controlling body orientation. However, a flexible sensory integration mechanism is missing from robot control leaving the robot vulnerable to instability in conditions where humans are able to maintain stance. We suggest the addition of a simple form of sensory integration to improve robot stance control. We also investigate how the biological constraint of feedback time delay influences the human stance control design. The human system may serve as a guide for improved robot control, but should not be directly copied because the constraints on robot and human control are different.     

Functional recovery following manipulation of muscles and sense organs in the stick insect leg

We studied functional recovery of leg posture and walking behaviour in the femur-tibia joint control system of stick insects. Leg extensions in resting animals and during walking are produced by different parts of a single extensor muscle. (a) Ablation of the muscle part responsible for fast movements prevented leg extension during the swing phase. Resting posture remained unaffected. Within a few post-operative days, extension movements recovered, provided that sensory feedback was available. Extension movements were now driven by the muscle part which in intact animals controls the resting posture only. (b) Selective ablation of this (slow) muscle part affected the resting posture, while walking was unaffected. The resting posture partly recovered during subsequent days. To test the range of functional recovery and underlying mechanisms, we additionally transected muscle motor innervation, or we inverted or ablated sensory feedback. We found that recovery was based on both muscular and neuronal mechanisms. The latter required appropriate sensory feedback for the process of recovery, but not for the maintenance of the recovered state. Our results thus indicate the existence of a sensory template that guides recovery. Recovery was limited to a behavioural range that occurs naturally in intact animals, though in different behavioural contexts.     

Sensory feedback in CNN-based central pattern generators

Central Pattern Generators (CPGs) are a suitable paradigm to solve the problem of locomotion control in walking robots. CPGs are able to generate feed-forward signals to achieve a proper coordination among the robot legs. In literature they are often modelled as networks of coupled nonlinear systems. However the topic of feedback in these systems is rarely addressed. On the other hand feedback is essential for locomotion. In this paper the CPG for a hexapod robot is implemented through Cellular Neural Networks (CNNs). Feedback is included in the CPG controller by exploiting the dynamic properties of the CPG motor-neurons, such as synchronization issue and local bifurcations. These universal paradigms provide the essential issues to include sensory feedback in CPG architectures based on coupled nonlinear systems. Experiments on a dynamic model of a hexapod robot are presented to validate the approach introduced.     

Regulation of motor circuit assembly by spatial and temporal mechanisms

Precision of synaptic connections in neuronal circuits is the product of an orderly assembly process during development. Circuits controlling motor behavior have been studied extensively in many animal species, allowing an assessment of evolutionarily conserved organizational principles that underlie neuronal subtype specification, connectivity and function. Across phylogenetically distant species, motor circuit assembly is based on spatial organization of interconnected circuit elements. Developmental molecular coordinate systems demarcate dendritic and axonal targeting territories, thereby regulating convergence of synaptic partners. Additional mechanisms subsequently control fine synaptic connection specificity within these domains. Spatial organization often correlates with the orderly sequence of neurogenesis contributing to the generation of distinct postmitotic neuronal subpopulations, a developmental strategy implemented far beyond motor circuits.     

Differential control of temporal and spatial aspects of cockroach leg coordination

Ensembles of neuronal networks and sensory pathways participate in controlling the kinematic and dynamic parameters of animal movement necessary to achieve motor coordination. Determining the relative contribution of proprioceptive feedback is essential for understanding how animals sustain stable, coordinated locomotion in complex natural environments. Here, we focus on the role of chordotonal organs (COs), proprioceptors found in insect legs, in the spatial and temporal regulation of walking. We compare gait parameters of intact cockroaches (Periplaneta americana) and sensory-impaired ones, injected with pymetrozine, a chemical previously shown to abolish CO function in locusts. We verify that afferent CO activity in pymetrozine-treated cockroaches is inhibited, and analyze the effect of this sensory deprivation on inter-leg coordination. We find significant changes in tarsi placement and leg path trajectories after pymetrozine treatment. Leg touchdown accuracy, measured from relative tarsi positions of adjacent legs, is reduced in treated animals. Interestingly, despite poorer spatial coordination in both stance and swing, temporal properties of the gait remain largely the same as in the intact preparations, apart from changes in ipsilateral phase differences between front and middle legs. These findings provide insights into the role of COs in insect gait control and establish pymetrozine as a useful tool for further studies of insect locomotion.     

Evolving coordinated group behaviours through maximisation of mean mutual information

A well known problem in the design of the control system for a swarm of robots concerns the definition of suitable individual rules that result in the desired coordinated behaviour. A possible solution to this problem is given by the automatic synthesis of the individual controllers through evolutionary or learning processes. These processes offer the possibility to freely search the space of the possible solutions for a given task, under the guidance of a user-defined utility function. Nonetheless, there exist no general principles to follow in the definition of such a utility function in order to reward coordinated group behaviours. As a consequence, task dependent functions must be devised each time a new coordination problem is under study. In this paper, we propose the use of measures developed in Information Theory as task-independent, implicit utility functions. We present two experiments in which three robots are trained to produce generic coordinated behaviours. Each robot is provided with rich sensory and motor apparatus, which can be exploited to explore the environment and to communicate with other robots. We show how coordinated behaviours can be synthesised through a simple evolutionary process. The only criteria used to evaluate the performance of the robotic group is the estimate of mutual information between the motor states of the robots.     

Similar sensorimotor transformations control balance during standing and walking

Standing and walking balance control in humans relies on the transformation of sensory information to motor commands that drive muscles. Here, we evaluated whether sensorimotor transformations underlying walking balance control can be described by task-level center of mass kinematics feedback similar to standing balance control. We found that delayed linear feedback of center of mass position and velocity, but not delayed linear feedback from ankle angles and angular velocities, can explain reactive ankle muscle activity and joint moments in response to perturbations of walking across protocols (discrete and continuous platform translations and discrete pelvis pushes). Feedback gains were modulated during the gait cycle and decreased with walking speed. Our results thus suggest that similar task-level variables, i.e. center of mass position and velocity, are controlled across standing and walking but that feedback gains are modulated during gait to accommodate changes in body configuration during the gait cycle and in stability with walking speed. These findings have important implications for modelling the neuromechanics of human balance control and for biomimetic control of wearable robotic devices. The feedback mechanisms we identified can be used to extend the current neuromechanical models that lack balance control mechanisms for the ankle joint. When using these models in the control of wearable robotic devices, we believe that this will facilitate shared control of balance between the user and the robotic device.     

Interference and Shaping in Sensorimotor Adaptations with Rewards

When a perturbation is applied in a sensorimotor transformation task, subjects can adapt and maintain performance by either relying on sensory feedback, or, in the absence of such feedback, on information provided by rewards. For example, in a classical rotation task where movement endpoints must be rotated to reach a fixed target, human subjects can successfully adapt their reaching movements solely on the basis of binary rewards, although this proves much more difficult than with visual feedback. Here, we investigate such a reward-driven sensorimotor adaptation process in a minimal computational model of the task. The key assumption of the model is that synaptic plasticity is gated by the reward. We study how the learning dynamics depend on the target size, the movement variability, the rotation angle and the number of targets. We show that when the movement is perturbed for multiple targets, the adaptation process for the different targets can interfere destructively or constructively depending on the similarities between the sensory stimuli (the targets) and the overlap in their neuronal representations. Destructive interferences can result in a drastic slowdown of the adaptation. As a result of interference, the time to adapt varies non-linearly with the number of targets. Our analysis shows that these interferences are weaker if the reward varies smoothly with the subject''s performance instead of being binary. We demonstrate how shaping the reward or shaping the task can accelerate the adaptation dramatically by reducing the destructive interferences. We argue that experimentally investigating the dynamics of reward-driven sensorimotor adaptation for more than one sensory stimulus can shed light on the underlying learning rules.     

Schema-based learning of adaptable and flexible prey-catching in anurans I. The basic architecture

A motor action often involves the coordination of several motor synergies and requires flexible adjustment of the ongoing execution based on feedback signals. To elucidate the neural mechanisms underlying the construction and selection of motor synergies, we study prey-capture in anurans. Experimental data demonstrate the intricate interaction between different motor synergies, including the interplay of their afferent feedback signals (Weerasuriya 1991; Anderson and Nishikawa 1996). Such data provide insights for the general issues concerning two-way information flow between sensory centers, motor circuits and periphery in motor coordination. We show how different afferent feedback signals about the status of the different components of the motor apparatus play a critical role in motor control as well as in learning. This paper, along with its companion paper, extend the model by Liaw et al. (1994) by integrating a number of different motor pattern generators, different types of afferent feedback, as well as the corresponding control structure within an adaptive framework we call Schema-Based Learning. We develop a model of the different MPGs involved in prey-catching as a vehicle to investigate the following questions: What are the characteristic features of the activity of a single muscle? How can these features be controlled by the premotor circuit? What are the strategies employed to generate and synchronize motor synergies? What is the role of afferent feedback in shaping the activity of a MPG? How can several MPGs share the same underlying circuitry and yet give rise to different motor patterns under different input conditions? In the companion paper we also extend the model by incorporating learning components that give rise to more flexible, adaptable and robust behaviors. To show these aspects we incorporate studies on experiments on lesions and the learning processes that allow the animal to recover its proper functioning     

Learning from sensory and reward prediction errors during motor adaptation

Voluntary motor commands produce two kinds of consequences. Initially, a sensory consequence is observed in terms of activity in our primary sensory organs (e.g., vision, proprioception). Subsequently, the brain evaluates the sensory feedback and produces a subjective measure of utility or usefulness of the motor commands (e.g., reward). As a result, comparisons between predicted and observed consequences of motor commands produce two forms of prediction error. How do these errors contribute to changes in motor commands? Here, we considered a reach adaptation protocol and found that when high quality sensory feedback was available, adaptation of motor commands was driven almost exclusively by sensory prediction errors. This form of learning had a distinct signature: as motor commands adapted, the subjects altered their predictions regarding sensory consequences of motor commands, and generalized this learning broadly to neighboring motor commands. In contrast, as the quality of the sensory feedback degraded, adaptation of motor commands became more dependent on reward prediction errors. Reward prediction errors produced comparable changes in the motor commands, but produced no change in the predicted sensory consequences of motor commands, and generalized only locally. Because we found that there was a within subject correlation between generalization patterns and sensory remapping, it is plausible that during adaptation an individual''s relative reliance on sensory vs. reward prediction errors could be inferred. We suggest that while motor commands change because of sensory and reward prediction errors, only sensory prediction errors produce a change in the neural system that predicts sensory consequences of motor commands.     

Adaptation to visual feedback delay influences visuomotor learning

Computational theory of motor control suggests that the brain continuously monitors motor commands, to predict their sensory consequences before actual sensory feedback becomes available. Such prediction error is a driving force of motor learning, and therefore appropriate associations between motor commands and delayed sensory feedback signals are crucial. Indeed, artificially introduced delays in visual feedback have been reported to degrade motor learning. However, considering our perceptual ability to causally bind our own actions with sensory feedback, demonstrated by the decrease in the perceived time delay following repeated exposure to an artificial delay, we hypothesized that such perceptual binding might alleviate deficits of motor learning associated with delayed visual feedback. Here, we evaluated this hypothesis by investigating the ability of human participants to adapt their reaching movements in response to a novel visuomotor environment with 3 visual feedback conditions--no-delay, sudden-delay, and adapted-delay. To introduce novelty into the trials, the cursor position, which originally indicated the hand position in baseline trials, was rotated around the starting position. In contrast to the no-delay condition, a 200-ms delay was artificially introduced between the cursor and hand positions during the presence of visual rotation (sudden-delay condition), or before the application of visual rotation (adapted-delay condition). We compared the learning rate (representing how the movement error modifies the movement direction in the subsequent trial) between the 3 conditions. In comparison with the no-delay condition, the learning rate was significantly degraded for the sudden-delay condition. However, this degradation was significantly alleviated by prior exposure to the delay (adapted-delay condition). Our data indicate the importance of appropriate temporal associations between motor commands and sensory feedback in visuomotor learning. Moreover, they suggest that the brain is able to account for such temporal associations in a flexible manner.     

Bifurcation and stability analysis in musculoskeletal systems: a study in human stance

Reflexes are important in the control of such daily activities as standing and walking. The goal of this study is to establish how reflexive feedback of muscle length, velocity, and force can lead to stable equilibria (i.e., posture) and limit cycles (e.g., ankle clonus and gait). The influence of stretch reflexes on the behavior and stability of musculoskeletal systems was examined using a model of human stance. We computed branches of fold and Hopf bifurcations by numerical bifurcation analysis of the model. These fold and Hopf branches divide the parameter space, constructed by the reflexive feedback gains, into regions of different behavior: unstable posture, stable posture, and stable limit cycles. These limit cycles correspond to a neural deficiency, termed ankle clonus. We also linked bifurcation analysis to known biomechanical concepts by linearizing the model: the fold branch corresponds to zero ankle stiffness and defines the minimal muscle length feedback necessary for stable posture; the Hopf branch is related to unstable reflex loops. Crossing the Hopf branch can lead to the above-mentioned stable limit cycles. The Hopf branch reduces with increasing time delays, making the subjects posture more susceptible to unstable reflex loops. This might be one of the reasons why elderly people, or those with injuries to the central nervous system, often have trouble with standing and other posture tasks. The influence of cocontraction and force feedback on the behavior of the posture model was also investigated. An increase in cocontraction leads to an increase in ankle stiffness (i.e., intrinsic muscle stiffness) and a decrease in the effective reflex loop gain. On the one hand, positive force feedback increases the ankle stiffness (i.e., intrinsic and reflexive muscle stiffness); on the other hand it makes the posture more susceptible to unstable reflex loops. For negative force feedback, the opposite is true. Finally, we calculated areas of reflex gains for perturbed stance and quiet stance in healthy subjects by fitting the model to data from the literature. The overlap of these areas of reflex gains could indicate that stretch reflexes are the major control mechanisms in both quiet and perturbed stance. In conclusion, this study has successfully combined bifurcation analysis with the more common biomechanical concepts and tools to determine the influence of reflexes on the stability and quality of stance. In the future, we will develop this line of research to look at rhythmic tasks, such as walking.     

A hierarchical neural-network model for control and learning of voluntary movement

In order to control voluntary movements, the central nervous system (CNS) must solve the following three computational problems at different levels: the determination of a desired trajectory in the visual coordinates, the transformation of its coordinates to the body coordinates and the generation of motor command. Based on physiological knowledge and previous models, we propose a hierarchical neural network model which accounts for the generation of motor command. In our model the association cortex provides the motor cortex with the desired trajectory in the body coordinates, where the motor command is then calculated by means of long-loop sensory feedback. Within the spinocerebellum — magnocellular red nucleus system, an internal neural model of the dynamics of the musculoskeletal system is acquired with practice, because of the heterosynaptic plasticity, while monitoring the motor command and the results of movement. Internal feedback control with this dynamical model updates the motor command by predicting a possible error of movement. Within the cerebrocerebellum — parvocellular red nucleus system, an internal neural model of the inverse-dynamics of the musculo-skeletal system is acquired while monitoring the desired trajectory and the motor command. The inverse-dynamics model substitutes for other brain regions in the complex computation of the motor command. The dynamics and the inverse-dynamics models are realized by a parallel distributed neural network, which comprises many sub-systems computing various nonlinear transformations of input signals and a neuron with heterosynaptic plasticity (that is, changes of synaptic weights are assumed proportional to a product of two kinds of synaptic inputs). Control and learning performance of the model was investigated by computer simulation, in which a robotic manipulator was used as a controlled system, with the following results: (1) Both the dynamics and the inverse-dynamics models were acquired during control of movements. (2) As motor learning proceeded, the inverse-dynamics model gradually took the place of external feedback as the main controller. Concomitantly, overall control performance became much better. (3) Once the neural network model learned to control some movement, it could control quite different and faster movements. (4) The neural netowrk model worked well even when only very limited information about the fundamental dynamical structure of the controlled system was available. Consequently, the model not only accounts for the learning and control capability of the CNS, but also provides a promising parallel-distributed control scheme for a large-scale complex object whose dynamics are only partially known.     

Neuronal control of respiration in decapod crustacea.

Respiratory exchange in decapod crustacea requires the coordinated activity of the heart and the scaphognathites, appendages which ventilate the gills. There is common central nervous system neuronal modulation of both autogenically active systems as well as direct neuronal communication between both systems. The heart and scaphognathites also respond directly to oxygen tension. The neuronal control of the scaphognathites also respond directly to oxygen tension. The neuronal control of the scaphognathites is analyzed at several levels. Particular attention is directed toward the means by which the innately organized and stereotyped motor pattern for forward beating can be altered to produce reversed beating. The importance of sensory feedback in maintaining normal rates of scaphognathite beating is noted. And the phenomenon of bilateral coordination between the morphologically independent scaphognathites is described. Several different models of parts of the over-all scaphognathite neuronal circuitry are presented for heuristic purposes.     

Neuromechanics: an integrative approach for understanding motor control

Neuromechanics seeks to understand how muscles, sense organs,motor pattern generators, and brain interact to produce coordinatedmovement, not only in complex terrain but also when confrontedwith unexpected perturbations. Applications of neuromechanicsinclude ameliorating human health problems (including prosthesisdesign and restoration of movement following brain or spinalcord injury), as well as the design, actuation and control ofmobile robots. In animals, coordinated movement emerges fromthe interplay among descending output from the central nervoussystem, sensory input from body and environment, muscle dynamics,and the emergent dynamics of the whole animal. The inevitablecoupling between neural information processing and the emergentmechanical behavior of animals is a central theme of neuromechanics.Fundamentally, motor control involves a series of transformationsof information, from brain and spinal cord to muscles to body,and back to brain. The control problem revolves around the specifictransfer functions that describe each transformation. The transferfunctions depend on the rules of organization and operationthat determine the dynamic behavior of each subsystem (i.e.,central processing, force generation, emergent dynamics, andsensory processing). In this review, we (1) consider the contributionsof muscles, (2) sensory processing, and (3) central networksto motor control, (4) provide examples to illustrate the interplayamong brain, muscles, sense organs and the environment in thecontrol of movement, and (5) describe advances in both roboticsand neuromechanics that have emerged from application of biologicalprinciples in robotic design. Taken together, these studiesdemonstrate that (1) intrinsic properties of muscle contributeto dynamic stability and control of movement, particularly immediatelyafter perturbations; (2) proprioceptive feedback reinforcesthese intrinsic self-stabilizing properties of muscle; (3) controlsystems must contend with inevitable time delays that can simplifyor complicate control; and (4) like most animals under a varietyof circumstances, some robots use a trial and error processto tune central feedforward control to emergent body dynamics.