Visually guided gait modifications for stepping over an obstacle: a bio-inspired approach

There is an increasing interest in conceiving robotic systems that are able to move and act in an unstructured and not predefined environment, for which autonomy and adaptability are crucial features. In nature, animals are autonomous biological systems, which often serve as bio-inspiration models, not only for their physical and mechanical properties, but also their control structures that enable adaptability and autonomy—for which learning is (at least) partially responsible. This work proposes a system which seeks to enable a quadruped robot to online learn to detect and to avoid stumbling on an obstacle in its path. The detection relies in a forward internal model that estimates the robot’s perceptive information by exploring the locomotion repetitive nature. The system adapts the locomotion in order to place the robot optimally before attempting to step over the obstacle, avoiding any stumbling. Locomotion adaptation is achieved by changing control parameters of a central pattern generator (CPG)-based locomotion controller. The mechanism learns the necessary alterations to the stride length in order to adapt the locomotion by changing the required CPG parameter. Both learning tasks occur online and together define a sensorimotor map, which enables the robot to learn to step over the obstacle in its path. Simulation results show the feasibility of the proposed approach.     

Distributed scalable multi-robot learning using particle swarm optimization

Designing effective behavioral controllers for mobile robots can be difficult and tedious; this process can be circumvented by using online learning techniques which allow robots to generate their own controllers online in an automated fashion. In multi-robot systems, robots operating in parallel can potentially learn at a much faster rate by sharing information amongst themselves. In this work, we use an adapted version of the Particle Swarm Optimization algorithm in order to accomplish distributed online robotic learning in groups of robots with access to only local information. The effectiveness of the learning technique on a benchmark task (generating high-performance obstacle avoidance behavior) is evaluated for robot groups of various sizes, with the maximum group size allowing each robot to individually contain and manage a single PSO particle. To increase the realism of the technique, different PSO neighborhoods based on limitations of real robotic communication are tested and compared in this scenario. We explore the effect of varying communication power for one of these communication-based PSO neighborhoods. To validate the effectiveness of these learning techniques, fully distributed online learning experiments are run using a group of 10 real robots, generating results which support the findings from our simulations.     

Fine-tuning and the stability of recurrent neural networks

A central criticism of standard theoretical approaches to constructing stable, recurrent model networks is that the synaptic connection weights need to be finely-tuned. This criticism is severe because proposed rules for learning these weights have been shown to have various limitations to their biological plausibility. Hence it is unlikely that such rules are used to continuously fine-tune the network in vivo. We describe a learning rule that is able to tune synaptic weights in a biologically plausible manner. We demonstrate and test this rule in the context of the oculomotor integrator, showing that only known neural signals are needed to tune the weights. We demonstrate that the rule appropriately accounts for a wide variety of experimental results, and is robust under several kinds of perturbation. Furthermore, we show that the rule is able to achieve stability as good as or better than that provided by the linearly optimal weights often used in recurrent models of the integrator. Finally, we discuss how this rule can be generalized to tune a wide variety of recurrent attractor networks, such as those found in head direction and path integration systems, suggesting that it may be used to tune a wide variety of stable neural systems.     

Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization

Brain-machine interface (BMI) systems give users direct neural control of robotic, communication, or functional electrical stimulation systems. As BMI systems begin transitioning from laboratory settings into activities of daily living, an important goal is to develop neural decoding algorithms that can be calibrated with a minimal burden on the user, provide stable control for long periods of time, and can be responsive to fluctuations in the decoder’s neural input space (e.g. neurons appearing or being lost amongst electrode recordings). These are significant challenges for static neural decoding algorithms that assume stationary input/output relationships. Here we use an actor-critic reinforcement learning architecture to provide an adaptive BMI controller that can successfully adapt to dramatic neural reorganizations, can maintain its performance over long time periods, and which does not require the user to produce specific kinetic or kinematic activities to calibrate the BMI. Two marmoset monkeys used the Reinforcement Learning BMI (RLBMI) to successfully control a robotic arm during a two-target reaching task. The RLBMI was initialized using random initial conditions, and it quickly learned to control the robot from brain states using only a binary evaluative feedback regarding whether previously chosen robot actions were good or bad. The RLBMI was able to maintain control over the system throughout sessions spanning multiple weeks. Furthermore, the RLBMI was able to quickly adapt and maintain control of the robot despite dramatic perturbations to the neural inputs, including a series of tests in which the neuron input space was deliberately halved or doubled.     

Interference and Shaping in Sensorimotor Adaptations with Rewards

When a perturbation is applied in a sensorimotor transformation task, subjects can adapt and maintain performance by either relying on sensory feedback, or, in the absence of such feedback, on information provided by rewards. For example, in a classical rotation task where movement endpoints must be rotated to reach a fixed target, human subjects can successfully adapt their reaching movements solely on the basis of binary rewards, although this proves much more difficult than with visual feedback. Here, we investigate such a reward-driven sensorimotor adaptation process in a minimal computational model of the task. The key assumption of the model is that synaptic plasticity is gated by the reward. We study how the learning dynamics depend on the target size, the movement variability, the rotation angle and the number of targets. We show that when the movement is perturbed for multiple targets, the adaptation process for the different targets can interfere destructively or constructively depending on the similarities between the sensory stimuli (the targets) and the overlap in their neuronal representations. Destructive interferences can result in a drastic slowdown of the adaptation. As a result of interference, the time to adapt varies non-linearly with the number of targets. Our analysis shows that these interferences are weaker if the reward varies smoothly with the subject''s performance instead of being binary. We demonstrate how shaping the reward or shaping the task can accelerate the adaptation dramatically by reducing the destructive interferences. We argue that experimentally investigating the dynamics of reward-driven sensorimotor adaptation for more than one sensory stimulus can shed light on the underlying learning rules.     

Hebbian learning in parallel and modular memories

Many cognitive and sensorimotor functions in the brain involve parallel and modular memory subsystems that are adapted by activity-dependent Hebbian synaptic plasticity. This is in contrast to the multilayer perceptron model of supervised learning where sensory information is presumed to be integrated by a common pool of hidden units through backpropagation learning. Here we show that Hebbian learning in parallel and modular memories is more advantageous than backpropagation learning in lumped memories in two respects: it is computationally much more efficient and structurally much simpler to implement with biological neurons. Accordingly, we propose a more biologically relevant neural network model, called a tree-like perceptron, which is a simple modification of the multilayer perceptron model to account for the general neural architecture, neuronal specificity, and synaptic learning rule in the brain. The model features a parallel and modular architecture in which adaptation of the input-to-hidden connection follows either a Hebbian or anti-Hebbian rule depending on whether the hidden units are excitatory or inhibitory, respectively. The proposed parallel and modular architecture and implicit interplay between the types of synaptic plasticity and neuronal specificity are exhibited by some neocortical and cerebellar systems. Received: 13 October 1996 / Accepted in revised form: 16 October 1997     

A burst-based "Hebbian" learning rule at retinogeniculate synapses links retinal waves to activity-dependent refinement

Patterned spontaneous activity in the developing retina is necessary to drive synaptic refinement in the lateral geniculate nucleus (LGN). Using perforated patch recordings from neurons in LGN slices during the period of eye segregation, we examine how such burst-based activity can instruct this refinement. Retinogeniculate synapses have a novel learning rule that depends on the latencies between pre- and postsynaptic bursts on the order of one second: coincident bursts produce long-lasting synaptic enhancement, whereas non-overlapping bursts produce mild synaptic weakening. It is consistent with “Hebbian” development thought to exist at this synapse, and we demonstrate computationally that such a rule can robustly use retinal waves to drive eye segregation and retinotopic refinement. Thus, by measuring plasticity induced by natural activity patterns, synaptic learning rules can be linked directly to their larger role in instructing the patterning of neural connectivity.     

Fly-like visuomotor responses of a robot using aVLSI motion-sensitive chips

 We explore the use of continuous-time analog very-large-scale-integrated (aVLSI) neuromorphic visual preprocessors together with a robotic platform in generating bio-inspired behaviors. Both the aVLSI motion sensors and the robot behaviors described in this work are inspired by the motion computation in the fly visual system and two different fly behaviors. In most robotic systems, the visual information comes from serially scanned imagers. This restricts the form of computation of the visual image and slows down the input rate to the controller system of the robot, hence increasing the reaction time of the robot. These aVLSI neuromorphic sensors reduce the computational load and power consumption of the robot, thus making it possible to explore continuous-time visuomotor control systems that react in real-time to the environment. The motion sensor provides two outputs: one for the preferred direction and the other for the null direction. These motion outputs are created from the aggregation of six elementary motion detectors that implement a variant of Reichardt's correlation algorithm. The four analog continuous-time outputs from the motion chips go to the control system on the robot which generates a mixture of two behaviors – course stabilization and fixation – from the outputs of these sensors. Since there are only four outputs, the amount of information transmitted to the controller is reduced (as compared to using a CCD sensor), and the reaction time of the robot is greatly decreased. In this work, the robot samples the motion sensors every 3.3 ms during the behavioral experiments. Received: 4 October 1999 / Accepted in revised form: 26 April 2001     

A Burst-Based “Hebbian” Learning Rule at Retinogeniculate Synapses Links Retinal Waves to Activity-Dependent Refinement

Patterned spontaneous activity in the developing retina is necessary to drive synaptic refinement in the lateral geniculate nucleus (LGN). Using perforated patch recordings from neurons in LGN slices during the period of eye segregation, we examine how such burst-based activity can instruct this refinement. Retinogeniculate synapses have a novel learning rule that depends on the latencies between pre- and postsynaptic bursts on the order of one second: coincident bursts produce long-lasting synaptic enhancement, whereas non-overlapping bursts produce mild synaptic weakening. It is consistent with “Hebbian” development thought to exist at this synapse, and we demonstrate computationally that such a rule can robustly use retinal waves to drive eye segregation and retinotopic refinement. Thus, by measuring plasticity induced by natural activity patterns, synaptic learning rules can be linked directly to their larger role in instructing the patterning of neural connectivity.     

Regulation of motor behaviour

The review is devoted to analysis of the basic links of motor behavior control systems: sensorimotor cortex, cerebellum, a red nucleus and striatum. The organization and communications of these structures and their participation in learning and memory processes are described. The synaptic neurotransmitter and nonsynaptic neuromodulatory systems innervating these structures are also described. Hierarchical synaptic networks are formed by GABA and glutamatergic systems. The nonsynaptic dopaminergic system innervates both of these structures, but carries out a modulatory function. The mesocorticolimbic dopaminergic system induces an emotional and motivational state - processes of reinforcement, and participates in realization of purposeful behavior. The nigrostriatal dopaminergic system, through triggering an endocellular signal and the processes ofphosphorylation and dephosphorylation modulates activity ofGABA and glutamatergic receptors ofdorsal striatum spiny neurons and adapted thalamocortical networks.     

Mykhailo F. Shuba

The spike activity of neurons of the sensorimotor cortex was analyzed in cats before, during, and after iontophoretic application of substances influencing synaptic transmission. It was shown that, in addition to glutamate ionotropic receptors, glutamate metabotropic receptors, as well as adrenergic and dopaminergic receptor systems, are involved in plastic rearrangements of synaptic connections among neocortical neurons. The pattern of aftereffects allows us to suppose that potentiation of synaptic events during conditioned reflex-related learning can be maintained for 10 to 20 min, an interval sufficient for consolidation of a memory track.     

The evolution of the human mind and logic--mathematics structures

The evolution of the human mind is discussed based on: (i) the fact that living beings interchange matter, energy and information with their environment, (ii) an ontological interpretation of the "reality" of the quantum world, of which logic-mathematics structures are considered constitutive parts, (iii) recent theories according to which living beings are considered as dynamic complex systems organized by information, and (iv) the fact that the evolution of living beings is guided by information about the environment and by intrinsic information on living systems (auto-organization). Assuming the evolution of vision as a model we observe that the driving forces that directed the evolution of the eyes, as dynamic complex systems, are the information about the environment supplied by sunlight and the intrinsic information-gaining mechanism of living organisms. Thus, there exists a convergence toward a visual system with the greatest ability to obtain light information, like the human eye, and also a divergence that leads to the development of specific qualities in some species. As in the case of vision the evolution of the human mind-brain cannot be a consequence of factors unrelated to the object of its own functioning. The human mind was structured for the acquisition from reality of the logic-mathematics structures that underlie the whole universe and consequently of an internal representation of the external world and of its own self. Thus, these structures are, together with the intrinsic capacity for auto-organization of the human brain, the predominant driving force of the human mind evolution. Both factors are complementary.     

Conditioned reflex-related plasticity in the neocortex

The spike activity of neurons of the sensorimotor cortex was analyzed in cats before, during, and after iontophoretic application of substances influencing synaptic transmission. It was shown that, in addition to glutamate ionotropic receptors, glutamate metabotropic receptors, as well as adrenergic and dopaminergic receptor systems, are involved in plastic rearrangements of synaptic connections among neocortical neurons. The pattern of aftereffects allows us to suppose that potentiation of synaptic events during conditioned reflex-related learning can be maintained for 10 to 20 min, an interval sufficient for consolidation of a memory track.     

Somato-dendritic Synaptic Plasticity and Error-backpropagation in Active Dendrites

In the last decade dendrites of cortical neurons have been shown to nonlinearly combine synaptic inputs by evoking local dendritic spikes. It has been suggested that these nonlinearities raise the computational power of a single neuron, making it comparable to a 2-layer network of point neurons. But how these nonlinearities can be incorporated into the synaptic plasticity to optimally support learning remains unclear. We present a theoretically derived synaptic plasticity rule for supervised and reinforcement learning that depends on the timing of the presynaptic, the dendritic and the postsynaptic spikes. For supervised learning, the rule can be seen as a biological version of the classical error-backpropagation algorithm applied to the dendritic case. When modulated by a delayed reward signal, the same plasticity is shown to maximize the expected reward in reinforcement learning for various coding scenarios. Our framework makes specific experimental predictions and highlights the unique advantage of active dendrites for implementing powerful synaptic plasticity rules that have access to downstream information via backpropagation of action potentials.     

A Bayesian Developmental Approach to Robotic Goal-Based Imitation Learning

A fundamental challenge in robotics today is building robots that can learn new skills by observing humans and imitating human actions. We propose a new Bayesian approach to robotic learning by imitation inspired by the developmental hypothesis that children use self-experience to bootstrap the process of intention recognition and goal-based imitation. Our approach allows an autonomous agent to: (i) learn probabilistic models of actions through self-discovery and experience, (ii) utilize these learned models for inferring the goals of human actions, and (iii) perform goal-based imitation for robotic learning and human-robot collaboration. Such an approach allows a robot to leverage its increasing repertoire of learned behaviors to interpret increasingly complex human actions and use the inferred goals for imitation, even when the robot has very different actuators from humans. We demonstrate our approach using two different scenarios: (i) a simulated robot that learns human-like gaze following behavior, and (ii) a robot that learns to imitate human actions in a tabletop organization task. In both cases, the agent learns a probabilistic model of its own actions, and uses this model for goal inference and goal-based imitation. We also show that the robotic agent can use its probabilistic model to seek human assistance when it recognizes that its inferred actions are too uncertain, risky, or impossible to perform, thereby opening the door to human-robot collaboration.     

Neurobiologically realistic determinants of self-organized criticality in networks of spiking neurons

Self-organized criticality refers to the spontaneous emergence of self-similar dynamics in complex systems poised between order and randomness. The presence of self-organized critical dynamics in the brain is theoretically appealing and is supported by recent neurophysiological studies. Despite this, the neurobiological determinants of these dynamics have not been previously sought. Here, we systematically examined the influence of such determinants in hierarchically modular networks of leaky integrate-and-fire neurons with spike-timing-dependent synaptic plasticity and axonal conduction delays. We characterized emergent dynamics in our networks by distributions of active neuronal ensemble modules (neuronal avalanches) and rigorously assessed these distributions for power-law scaling. We found that spike-timing-dependent synaptic plasticity enabled a rapid phase transition from random subcritical dynamics to ordered supercritical dynamics. Importantly, modular connectivity and low wiring cost broadened this transition, and enabled a regime indicative of self-organized criticality. The regime only occurred when modular connectivity, low wiring cost and synaptic plasticity were simultaneously present, and the regime was most evident when between-module connection density scaled as a power-law. The regime was robust to variations in other neurobiologically relevant parameters and favored systems with low external drive and strong internal interactions. Increases in system size and connectivity facilitated internal interactions, permitting reductions in external drive and facilitating convergence of postsynaptic-response magnitude and synaptic-plasticity learning rate parameter values towards neurobiologically realistic levels. We hence infer a novel association between self-organized critical neuronal dynamics and several neurobiologically realistic features of structural connectivity. The central role of these features in our model may reflect their importance for neuronal information processing.     

Brain plasticity and ion channels

It is generally believed that spatio-temporal configurations of distributed activity in the brain contribute to the coding of neuronal information and that synaptic contacts between nerve cells could play a central role in the formation of privileged pathways of activity. Synaptic plasticity is not the only mode of regulation of information processing in the brain and persistent regulations of ionic conductances in some specialized neuronal areas such as the dendrites, the cell body and the axon could also modulate, in the short- and the long-term, the propagation of information in the brain. Persistent changes in intrinsic excitability have been reported in several brain areas in which activity is modified during a classical conditioning. The role of synaptic activity seems to be determinant in the induction but the learning rules and the underlying mechanisms remain to be defined. This review discusses the role of neuronal activity in the induction of intrinsic plasticity in cortical, hippocampal and cerebellar neurons. Activation and inactivation properties of ionic channels in the axon determine the short-term dynamics of axonal propagation and synaptic transmission. Activation of glutamate receptors initiates a long-term modification in neuronal excitability that may represent the substrate for the mnesic engram and for the stabilization of the epileptic state. Similarly to synaptic plasticity, long-lasting intrinsic plasticity appears to be reversible and to express a certain level of input or cellular specificity. These non-synaptic forms of plasticity affect the signal propagation in the axon, the dendrites and the soma. They not only share common learning rules and induction pathways with the better known synaptic plasticity such as NMDA receptor-dependent LTP and LTD but also contribute in synergy with these synaptic changes to the formation of a coherent mnesic engram.     

The chronotron: a neuron that learns to fire temporally precise spike patterns

In many cases, neurons process information carried by the precise timings of spikes. Here we show how neurons can learn to generate specific temporally precise output spikes in response to input patterns of spikes having precise timings, thus processing and memorizing information that is entirely temporally coded, both as input and as output. We introduce two new supervised learning rules for spiking neurons with temporal coding of information (chronotrons), one that provides high memory capacity (E-learning), and one that has a higher biological plausibility (I-learning). With I-learning, the neuron learns to fire the target spike trains through synaptic changes that are proportional to the synaptic currents at the timings of real and target output spikes. We study these learning rules in computer simulations where we train integrate-and-fire neurons. Both learning rules allow neurons to fire at the desired timings, with sub-millisecond precision. We show how chronotrons can learn to classify their inputs, by firing identical, temporally precise spike trains for different inputs belonging to the same class. When the input is noisy, the classification also leads to noise reduction. We compute lower bounds for the memory capacity of chronotrons and explore the influence of various parameters on chronotrons' performance. The chronotrons can model neurons that encode information in the time of the first spike relative to the onset of salient stimuli or neurons in oscillatory networks that encode information in the phases of spikes relative to the background oscillation. Our results show that firing one spike per cycle optimizes memory capacity in neurons encoding information in the phase of firing relative to a background rhythm.     

LTD windows of the STDP learning rule and synaptic connections having a large transmission delay enable robust sequence learning amid background noise

Spike-timing-dependent synaptic plasticity (STDP) is a simple and effective learning rule for sequence learning. However, synapses being subject to STDP rules are readily influenced in noisy circumstances because synaptic conductances are modified by pre- and postsynaptic spikes elicited within a few tens of milliseconds, regardless of whether those spikes convey information or not. Noisy firing existing everywhere in the brain may induce irrelevant enhancement of synaptic connections through STDP rules and would result in uncertain memory encoding and obscure memory patterns. We will here show that the LTD windows of the STDP rules enable robust sequence learning amid background noise in cooperation with a large signal transmission delay between neurons and a theta rhythm, using a network model of the entorhinal cortex layer II with entorhinal-hippocampal loop connections. The important element of the present model for robust sequence learning amid background noise is the symmetric STDP rule having LTD windows on both sides of the LTP window, in addition to the loop connections having a large signal transmission delay and the theta rhythm pacing activities of stellate cells. Above all, the LTD window in the range of positive spike-timing is important to prevent influences of noise with the progress of sequence learning.     

Stability of complex spike timing-dependent plasticity in cerebellar learning

Dynamics of spike-timing dependent synaptic plasticity are analyzed for excitatory and inhibitory synapses onto cerebellar Purkinje cells. The purpose of this study is to place theoretical constraints on candidate synaptic learning rules that determine the changes in synaptic efficacy due to pairing complex spikes with presynaptic spikes in parallel fibers and inhibitory interneurons. Constraints are derived for the timing between complex spikes and presynaptic spikes, constraints that result from the stability of the learning dynamics of the learning rule. Potential instabilities in the parallel fiber synaptic learning rule are found to be stabilized by synaptic plasticity at inhibitory synapses if the inhibitory learning rules are stable, and conditions for stability of inhibitory plasticity are given. Combining excitatory with inhibitory plasticity provides a mechanism for minimizing the overall synaptic input. Stable learning rules are shown to be able to sculpt simple-spike patterns by regulating the excitability of neurons in the inferior olive that give rise to climbing fibers.