Learning visuomotor transformations for gaze-control and grasping

For reaching to and grasping of an object, visual information about the object must be transformed into motor or postural commands for the arm and hand. In this paper, we present a robot model for visually guided reaching and grasping. The model mimics two alternative processing pathways for grasping, which are also likely to coexist in the human brain. The first pathway directly uses the retinal activation to encode the target position. In the second pathway, a saccade controller makes the eyes (cameras) focus on the target, and the gaze direction is used instead as positional input. For both pathways, an arm controller transforms information on the target’s position and orientation into an arm posture suitable for grasping. For the training of the saccade controller, we suggest a novel staged learning method which does not require a teacher that provides the necessary motor commands. The arm controller uses unsupervised learning: it is based on a density model of the sensor and the motor data. Using this density, a mapping is achieved by completing a partially given sensorimotor pattern. The controller can cope with the ambiguity in having a set of redundant arm postures for a given target. The combined model of saccade and arm controller was able to fixate and grasp an elongated object with arbitrary orientation and at arbitrary position on a table in 94% of trials.     

Motor learning by observing

Learning complex motor behaviors like riding a bicycle or swinging a golf club is based on acquiring neural representations of the mechanical requirements of movement (e.g., coordinating muscle forces to control the club). Here we provide evidence that mechanisms matching observation and action facilitate motor learning. Subjects who observed a video depicting another person learning to reach in a novel mechanical environment (imposed by a robot arm) performed better when later tested in the same environment than subjects who observed similar movements but no learning; moreover, subjects who observed learning of a different environment performed worse. We show that this effect is not based on conscious strategies but instead depends on the implicit engagement of neural systems for movement planning and control.     

Computational nature of human adaptive control during learning of reaching movements in force fields

Learning to make reaching movements in force fields was used as a paradigm to explore the system architecture of the biological adaptive controller. We compared the performance of a number of candidate control systems that acted on a model of the neuromuscular system of the human arm and asked how well the dynamics of the candidate system compared with the movement characteristics of 16 subjects. We found that control via a supra-spinal system that utilized an adaptive inverse model resulted in dynamics that were similar to that observed in our subjects, but lacked essential characteristics. These characteristics pointed to a different architecture where descending commands were influenced by an adaptive forward model. However, we found that control via a forward model alone also resulted in dynamics that did not match the behavior of the human arm. We considered a third control architecture where a forward model was used in conjunction with an inverse model and found that the resulting dynamics were remarkably similar to that observed in the experimental data. The essential property of this control architecture was that it predicted a complex pattern of near-discontinuities in hand trajectory in the novel force field. A nearly identical pattern was observed in our subjects, suggesting that generation of descending motor commands was likely through a control system architecture that included both adaptive forward and inverse models. We found that as subjects learned to make reaching movements, adaptation rates for the forward and inverse models could be independently estimated and the resulting changes in performance of subjects from movement to movement could be accurately accounted for. Results suggested that the adaptation of the forward model played a dominant role in the motor learning of subjects. After a period of consolidation, the rates of adaptation in the internal models were significantly larger than those observed before the memory had consolidated. This suggested that consolidation of motor memory coincided with freeing of certain computational resources for subsequent learning. Received: 01 January 1998 / Accepted in revised form: 26 January 1999     

Why Robots Should Be Social: Enhancing Machine Learning through Social Human-Robot Interaction

Social learning is a powerful method for cultural propagation of knowledge and skills relying on a complex interplay of learning strategies, social ecology and the human propensity for both learning and tutoring. Social learning has the potential to be an equally potent learning strategy for artificial systems and robots in specific. However, given the complexity and unstructured nature of social learning, implementing social machine learning proves to be a challenging problem. We study one particular aspect of social machine learning: that of offering social cues during the learning interaction. Specifically, we study whether people are sensitive to social cues offered by a learning robot, in a similar way to children’s social bids for tutoring. We use a child-like social robot and a task in which the robot has to learn the meaning of words. For this a simple turn-based interaction is used, based on language games. Two conditions are tested: one in which the robot uses social means to invite a human teacher to provide information based on what the robot requires to fill gaps in its knowledge (i.e. expression of a learning preference); the other in which the robot does not provide social cues to communicate a learning preference. We observe that conveying a learning preference through the use of social cues results in better and faster learning by the robot. People also seem to form a “mental model” of the robot, tailoring the tutoring to the robot’s performance as opposed to using simply random teaching. In addition, the social learning shows a clear gender effect with female participants being responsive to the robot’s bids, while male teachers appear to be less receptive. This work shows how additional social cues in social machine learning can result in people offering better quality learning input to artificial systems, resulting in improved learning performance.     

Moving just like you: motor interference depends on similar motility of agent and observer

Recent findings in neuroscience suggest an overlap between brain regions involved in the execution of movement and perception of another's movement. This so-called "action-perception coupling" is supposed to serve our ability to automatically infer the goals and intentions of others by internal simulation of their actions. A consequence of this coupling is motor interference (MI), the effect of movement observation on the trajectory of one's own movement. Previous studies emphasized that various features of the observed agent determine the degree of MI, but could not clarify how human-like an agent has to be for its movements to elicit MI and, more importantly, what 'human-like' means in the context of MI. Thus, we investigated in several experiments how different aspects of appearance and motility of the observed agent influence motor interference (MI). Participants performed arm movements in horizontal and vertical directions while observing videos of a human, a humanoid robot, or an industrial robot arm with either artificial (industrial) or human-like joint configurations. Our results show that, given a human-like joint configuration, MI was elicited by observing arm movements of both humanoid and industrial robots. However, if the joint configuration of the robot did not resemble that of the human arm, MI could longer be demonstrated. Our findings present evidence for the importance of human-like joint configuration rather than other human-like features for perception-action coupling when observing inanimate agents.     

A neural network model for the intersensory coordination involved in goal-directed movements

A neural network model for a sensorimotor system, which was developed to simulate oriented movements in man, is presented. It is composed of a formal neural network comprising two layers: a sensory layer receiving and processing sensory inputs, and a motor layer driving a simulated arm. The sensory layer is an extension of the topological network previously proposed by Kohonen (1984). Two kinds of sensory modality, proprioceptive and exteroceptive, are used to define the arm position. Each sensory cell receives proprioceptive inputs provided by each arm-joint together with the exteroceptive inputs. This sensory layer is therefore a kind of associative layer which integrates two separate sensory signals relating to movement coding. It is connected to the motor layer by means of adaptive synapses which provide a physical link between a motor activity and its sensory consequences. After a learning period, the spatial map which emerges in the sensory layer clearly depends on the sensory inputs and an associative map of both the arm and the extra-personal space is built up if proprioceptive and exteroceptive signals are processed together. The sensorimotor transformations occuring in the junctions linking the sensory and motor layers are organized in such a manner that the simulated arm becomes able to reach towards and track a target in extra-personal space. Proprioception serves to determine the final arm posture adopted and to correct the ongoing movement in cases where changes in the target location occur. With a view to developing a sensorimotor control system with more realistic salient features, a robotic model was coupled with the formal neural network. This robotic implementation of our model shows the capacity of formal neural networks to control the displacement of mechanical devices.     

A model of activity-dependent formation of cerebellar microzones

According to modern views of the cerebellum in motor control, each cerebellar functional unit, or microzone, learns how to execute predictive and coordinative control, based on long-term depression of the granule cell-Purkinje cell synapses. In the present paper, in light of recent experimental and theoretical studies on synaptic elimination and cerebellar motor learning, a model of the formation of cerebellar microzones by climbing fiber synaptic elimination is proposed. It is shown that competition for an activity-dependent supply of neurotrophic factor can reproduce the spatio-temporal characteristics of climbing fiber synaptic elimination. It is further shown that when this elimination is accurate, motor coordination can be acquired in an arm reaching task. In view of the results of the present study, several predictions are proposed. Received: 19 January 1998 / Accepted in revised form: 22 April 1998     

A mathematical model of the adaptive control of human arm motions

This paper discusses similarities between models of adaptive motor control suggested by recent experiments with human and animal subjects, and the structure of a new control law derived mathematically from nonlinear stability theory. In both models, the control actions required to track a specified trajectory are adaptively assembled from a large collection of simple computational elements. By adaptively recombining these elements, the controllers develop complex internal models which are used to compensate for the effects of externally imposed forces or changes in the physical properties of the system. On a motor learning task involving planar, multi-joint arm motions, the simulated performance of the mathematical model is shown to be qualitatively similar to observed human performance, suggesting that the model captures some of the interesting features of the dynamics of low-level motor adaptation. Received: 20 September 1994 / Accepted in revised form: 18 November 1998     

Learning to control a brain-machine interface for reaching and grasping by primates

Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain–machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.     

Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates

Reaching and grasping in primates depend on the coordination of neural activity in large frontoparietal ensembles. Here we demonstrate that primates can learn to reach and grasp virtual objects by controlling a robot arm through a closed-loop brain–machine interface (BMIc) that uses multiple mathematical models to extract several motor parameters (i.e., hand position, velocity, gripping force, and the EMGs of multiple arm muscles) from the electrical activity of frontoparietal neuronal ensembles. As single neurons typically contribute to the encoding of several motor parameters, we observed that high BMIc accuracy required recording from large neuronal ensembles. Continuous BMIc operation by monkeys led to significant improvements in both model predictions and behavioral performance. Using visual feedback, monkeys succeeded in producing robot reach-and-grasp movements even when their arms did not move. Learning to operate the BMIc was paralleled by functional reorganization in multiple cortical areas, suggesting that the dynamic properties of the BMIc were incorporated into motor and sensory cortical representations.     

Manuo-ocular coordination in target tracking. II. Comparing the model with human behavior

Several studies have shown that humans track a moving visual target with their eyes better if the movement of this target is directly controlled by the observer's hand. The improvement in performance has been attributed to coordination control between the arm motor system and the smooth pursuit (SP) system. In such a task, the SP system shows characteristics that differ from those observed during eye-alone tracking: latency (between the target-arm and the eye motion onsets) is shorter, maximum SP velocity is higher and the maximum target motion frequency at which the SP can function effectively is also higher. The aim of this article is to qualitatively evaluate the behavior of a dynamical model simulating the oculomotor system and the arm motor system when both are involved in tracking visual targets. The evaluation is essentially based on a comparison of the behavior of the model with the behavior of human subjects tracking visual targets under different conditions. The model has been introduced and quantitatively evaluated in a companion paper. The model is based on an exchange of internal information between the two sensorimotor systems, mediated by sensory signals (vision, arm muscle proprioception) and motor signals (arm motor command copy). The exchange is achieved by a specialized structure of the central nervous system, previously identified as a part of the cerebellum. Computer simulation of the model yielded results that fit the behavior of human subjects observed during previously reported experiments, both qualitatively and quantitatively. The parallelism between physiology and human behavior on the one hand, and structure and simulation of the model on the other hand, is discussed. Received: 6 March 1997 / Accepted in revised form: 15 July 1997     

A model for learning human reaching movements

 Reaching movement is a fast movement towards a given target. The main characteristics of such a movement are straight path and a bell-shaped speed profile. In this work a mathematical model for the control of the human arm during ballistic reaching movements is presented. The model of the arm contains a 2 degrees of freedom planar manipulator, and a Hill-type, non-linear mechanical model of six muscles. The arm model is taken from the literature with minor changes. The nervous system is modeled as an adjustable pattern generator that creates the control signals to the muscles. The control signals in this model are rectangular pulses activated at various amplitudes and timings, that are determined according to the given target. These amplitudes and timings are the parameters that should be related to each target and initial conditions in the workspace. The model of the nervous system consists of an artificial neural net that maps any given target to the parameter space of the pattern generator. In order to train this net, the nervous system model includes a sensitivity model that transforms the error from the arm end-point coordinates to the parameter coordinates. The error is assessed only at the termination of the movement from knowledge of the results. The role of the non-linearity in the muscle model and the performance of the learning scheme are analysed, illustrated in simulations and discussed. The results of the present study demonstrate the central nervous system’s (CNS) ability to generate typical reaching movements with a simple feedforward controller that controls only the timing and amplitude of rectangular excitation pulses to the muscles and adjusts these parameters based on knowledge of the results. In this scheme, which is based on the adjustment of only a few parameters instead of the whole trajectory, the dimension of the control problem is reduced significantly. It is shown that the non-linear properties of the muscles are essential to achieve this simple control. This conclusion agrees with the general concept that motor control is the result of an interaction between the nervous system and the musculoskeletal dynamics. Received : 21 May 1996 / Accepted in revised form : 10 June 1997     

Adaptive Robotic Control Driven by a Versatile Spiking Cerebellar Network

The cerebellum is involved in a large number of different neural processes, especially in associative learning and in fine motor control. To develop a comprehensive theory of sensorimotor learning and control, it is crucial to determine the neural basis of coding and plasticity embedded into the cerebellar neural circuit and how they are translated into behavioral outcomes in learning paradigms. Learning has to be inferred from the interaction of an embodied system with its real environment, and the same cerebellar principles derived from cell physiology have to be able to drive a variety of tasks of different nature, calling for complex timing and movement patterns. We have coupled a realistic cerebellar spiking neural network (SNN) with a real robot and challenged it in multiple diverse sensorimotor tasks. Encoding and decoding strategies based on neuronal firing rates were applied. Adaptive motor control protocols with acquisition and extinction phases have been designed and tested, including an associative Pavlovian task (Eye blinking classical conditioning), a vestibulo-ocular task and a perturbed arm reaching task operating in closed-loop. The SNN processed in real-time mossy fiber inputs as arbitrary contextual signals, irrespective of whether they conveyed a tone, a vestibular stimulus or the position of a limb. A bidirectional long-term plasticity rule implemented at parallel fibers-Purkinje cell synapses modulated the output activity in the deep cerebellar nuclei. In all tasks, the neurorobot learned to adjust timing and gain of the motor responses by tuning its output discharge. It succeeded in reproducing how human biological systems acquire, extinguish and express knowledge of a noisy and changing world. By varying stimuli and perturbations patterns, real-time control robustness and generalizability were validated. The implicit spiking dynamics of the cerebellar model fulfill timing, prediction and learning functions.     

Cerebellar learning of accurate predictive control for fast-reaching movements

Long conduction delays in the nervous system prevent the accurate control of movements by feedback control alone. We present a new, biologically plausible cerebellar model to study how fast arm movements can be executed in spite of these delays. To provide a realistic test-bed of the cerebellar neural model, we embed the cerebellar network in a simulated biological motor system comprising a spinal cord model and a six-muscle two-dimensional arm model. We argue that if the trajectory errors are detected at the spinal cord level, memory traces in the cerebellum can solve the temporal mismatch problem between efferent motor commands and delayed error signals. Moreover, learning is made stable by the inclusion of the cerebello-nucleo-olivary loop in the model. It is shown that the cerebellar network implements a nonlinear predictive regulator by learning part of the inverse dynamics of the plant and spinal circuit. After learning, fast accurate reaching movements can be generated. Received: 8 February 1999 /Accepted in revised form: 7 August 1999     

Impedance characteristics of a neuromusculoskeletal model of the human arm II. Movement control

The modulation of neuromusculoskeletal impedance during movements is analysed using a motor control model of the human arm. The motor control system combines feedback and feedforward control and both control modes are determined in one optimization process. In the model, the stiffness varies at the double movement frequency for 2-Hz oscillatory elbow movements and has high values at the movement reversals. During goal-directed two-degrees-of-freedom arm movements, the stiffness is decreased during the movement and may be increased in the initial and final phases, depending on the movement velocity. The stiffness has a considerable curl during the movement, as was also observed in experimental data. The dynamic stiffness patterns of the model can be explained basically by the α−γ coactivation scheme where feedback gains covary with motor control signals. In addition to the modulation of the gain factors, it is argued that the variation of the intrinsic stiffness has a considerable effect on movement control, especially during fast movements. Received: 14 October 1997 / Accepted in revised form: 18 May 1999     

A Novel Robot System Integrating Biological and Mechanical Intelligence Based on Dissociated Neural Network-Controlled Closed-Loop Environment

We propose the architecture of a novel robot system merging biological and artificial intelligence based on a neural controller connected to an external agent. We initially built a framework that connected the dissociated neural network to a mobile robot system to implement a realistic vehicle. The mobile robot system characterized by a camera and two-wheeled robot was designed to execute the target-searching task. We modified a software architecture and developed a home-made stimulation generator to build a bi-directional connection between the biological and the artificial components via simple binomial coding/decoding schemes. In this paper, we utilized a specific hierarchical dissociated neural network for the first time as the neural controller. Based on our work, neural cultures were successfully employed to control an artificial agent resulting in high performance. Surprisingly, under the tetanus stimulus training, the robot performed better and better with the increasement of training cycle because of the short-term plasticity of neural network (a kind of reinforced learning). Comparing to the work previously reported, we adopted an effective experimental proposal (i.e. increasing the training cycle) to make sure of the occurrence of the short-term plasticity, and preliminarily demonstrated that the improvement of the robot’s performance could be caused independently by the plasticity development of dissociated neural network. This new framework may provide some possible solutions for the learning abilities of intelligent robots by the engineering application of the plasticity processing of neural networks, also for the development of theoretical inspiration for the next generation neuro-prostheses on the basis of the bi-directional exchange of information within the hierarchical neural networks.     

Impedance characteristics of a neuromusculoskeletal model of the human arm I. Posture control

The mechanical impedance of neuromusculoskeletal models of the human arm is studied in this paper. The model analysis provides a better understanding of the contributions of possible intrinsic and reflexive components of arm impedance, makes clear the limitations of second-order mass-viscosity-stiffness models and reveals possible task effects on the impedance. The musculoskeletal model describes planar movements of the upper arm and forearm, which are moved by six lumped muscles with nonlinear dynamics. The motor control system is represented by a neural network which combines feedforward and feedback control. It is optimized for the control of movements or for posture control in the presence of external forces. The achieved impedance characteristics depend on the conditions during the learning process. In particular, the impedance is adapted in a suitable way to the frequency content and direction of external forces acting on the hand during an isometric task. The impedance characteristics of a model, which is optimized for movement control, are similar to experimental data in the literature. The achieved stiffness is, to a large extent, reflexively determined whereas the approximated viscosity is primarily due to intrinsic attributes. It is argued that usually applied Hill-type muscle models do not properly represent intrinsic muscle stiffness. Received: 14 October 1997 / Accepted in revised form: 18 May 1999     

Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses

 Some characteristics of arm movements that humans exhibit during learning the dynamics of reaching are consistent with a theoretical framework where training results in motor commands that are gradually modified to predict and compensate for novel forces that may act on the hand. As a first approximation, the motor control system behaves as an adapting controller that learns an internal model of the dynamics of the task. It approximates inverse dynamics and predicts motor commands that are appropriate for a desired limb trajectory. However, we had previously noted that subtle motion characteristics observed during changes in task dynamics challenged this simple model and raised the possibility that adaptation also involved sensory–motor feedback pathways. These pathways reacted to sensory feedback during the course of the movement. Here we hypothesize that adaptation to dynamics might also involve a modification of how the CNS responds to sensory feedback. We tested this through experiments that quantified how the motor system's response to errors during voluntary movements changed as it adapted to dynamics of a force field. We describe a nonlinear approach that approximates the impedance of the arm, i.e., force response as a function of arm displacement trajectory. We observe that after adaptation, the impedance function changes in a way that closely matches and counters the effect of the force field. This is particularly prominent in the long-latency (>100 ms) component of response to perturbations. Therefore, it appears that practice not only modifies the internal model with which the brain generates motor commands that initiate a movement, but also the internal model with which sensory feedback is integrated with the ongoing descending commands in order to respond to error during the movement. Received: 10 January 2001 / Accepted in revised form: 30 May 2001     

Stroke rehabilitation reaches a threshold

Motor training with the upper limb affected by stroke partially reverses the loss of cortical representation after lesion and has been proposed to increase spontaneous arm use. Moreover, repeated attempts to use the affected hand in daily activities create a form of practice that can potentially lead to further improvement in motor performance. We thus hypothesized that if motor retraining after stroke increases spontaneous arm use sufficiently, then the patient will enter a virtuous circle in which spontaneous arm use and motor performance reinforce each other. In contrast, if the dose of therapy is not sufficient to bring spontaneous use above threshold, then performance will not increase and the patient will further develop compensatory strategies with the less affected hand. To refine this hypothesis, we developed a computational model of bilateral hand use in arm reaching to study the interactions between adaptive decision making and motor relearning after motor cortex lesion. The model contains a left and a right motor cortex, each controlling the opposite arm, and a single action choice module. The action choice module learns, via reinforcement learning, the value of using each arm for reaching in specific directions. Each motor cortex uses a neural population code to specify the initial direction along which the contralateral hand moves towards a target. The motor cortex learns to minimize directional errors and to maximize neuronal activity for each movement. The derived learning rule accounts for the reversal of the loss of cortical representation after rehabilitation and the increase of this loss after stroke with insufficient rehabilitation. Further, our model exhibits nonlinear and bistable behavior: if natural recovery, motor training, or both, brings performance above a certain threshold, then training can be stopped, as the repeated spontaneous arm use provides a form of motor learning that further bootstraps performance and spontaneous use. Below this threshold, motor training is "in vain": there is little spontaneous arm use after training, the model exhibits learned nonuse, and compensatory movements with the less affected hand are reinforced. By exploring the nonlinear dynamics of stroke recovery using a biologically plausible neural model that accounts for reversal of the loss of motor cortex representation following rehabilitation or the lack thereof, respectively, we can explain previously hard to reconcile data on spontaneous arm use in stroke recovery. Further, our threshold prediction could be tested with an adaptive train-wait-train paradigm: if spontaneous arm use has increased in the "wait" period, then the threshold has been reached, and rehabilitation can be stopped. If spontaneous arm use is still low or has decreased, then another bout of rehabilitation is to be provided.     

Modelling of direct motor program learning in fast human arm motions

We propose and simulate a new paradigm for organization of motor control in fast and accurate human arm motions. We call the paradigm direct motor program learning since the control programs are learned directly without knowing or learning the dynamics of a controlled system.The idea is to approximate the dependence of the motor control programs on the vector of the task parameters rather than to use a model of the system dynamics. We apply iterative learning control and scattered data multivariate approximation techniques to achieve the goal. The advantage of the paradigm is that the control complexity depends neither on the order nor on the nonlinearity of the system dynamics.We simulate the direct motor program learning paradigm in the task of point-to-point control of fast planar human arm motions. Simulation takes into account nonlinear arm dynamics, muscle force dynamics, delay in low-level reflex feedback, time dependence of the feedback gains and coactivation of antagonist muscles. Despite highly nonlinear time-variant dynamics of the controlled system, reasonably good motion precision is obtained over a wide range of the task parameters (initial and final positions of the arm). The simulation results demonstrate that the paradigm is indeed viable and could be considered as a possible explanation for the organization of motor control of fast motions.