A real-time spiking cerebellum model for learning robot control

We describe a neural network model of the cerebellum based on integrate-and-fire spiking neurons with conductance-based synapses. The neuron characteristics are derived from our earlier detailed models of the different cerebellar neurons. We tested the cerebellum model in a real-time control application with a robotic platform. Delays were introduced in the different sensorimotor pathways according to the biological system. The main plasticity in the cerebellar model is a spike-timing dependent plasticity (STDP) at the parallel fiber to Purkinje cell connections. This STDP is driven by the inferior olive (IO) activity, which encodes an error signal using a novel probabilistic low frequency model. We demonstrate the cerebellar model in a robot control system using a target-reaching task. We test whether the system learns to reach different target positions in a non-destructive way, therefore abstracting a general dynamics model. To test the system's ability to self-adapt to different dynamical situations, we present results obtained after changing the dynamics of the robotic platform significantly (its friction and load). The experimental results show that the cerebellar-based system is able to adapt dynamically to different contexts.     

Adaptive Robotic Control Driven by a Versatile Spiking Cerebellar Network

The cerebellum is involved in a large number of different neural processes, especially in associative learning and in fine motor control. To develop a comprehensive theory of sensorimotor learning and control, it is crucial to determine the neural basis of coding and plasticity embedded into the cerebellar neural circuit and how they are translated into behavioral outcomes in learning paradigms. Learning has to be inferred from the interaction of an embodied system with its real environment, and the same cerebellar principles derived from cell physiology have to be able to drive a variety of tasks of different nature, calling for complex timing and movement patterns. We have coupled a realistic cerebellar spiking neural network (SNN) with a real robot and challenged it in multiple diverse sensorimotor tasks. Encoding and decoding strategies based on neuronal firing rates were applied. Adaptive motor control protocols with acquisition and extinction phases have been designed and tested, including an associative Pavlovian task (Eye blinking classical conditioning), a vestibulo-ocular task and a perturbed arm reaching task operating in closed-loop. The SNN processed in real-time mossy fiber inputs as arbitrary contextual signals, irrespective of whether they conveyed a tone, a vestibular stimulus or the position of a limb. A bidirectional long-term plasticity rule implemented at parallel fibers-Purkinje cell synapses modulated the output activity in the deep cerebellar nuclei. In all tasks, the neurorobot learned to adjust timing and gain of the motor responses by tuning its output discharge. It succeeded in reproducing how human biological systems acquire, extinguish and express knowledge of a noisy and changing world. By varying stimuli and perturbations patterns, real-time control robustness and generalizability were validated. The implicit spiking dynamics of the cerebellar model fulfill timing, prediction and learning functions.     

Synaptic conditions for auto-associative memory storage and pattern completion in Jensen et al.’s model of hippocampal area CA3

Jensen et al. (Learn Memory 3(2–3):243–256, 1996b) proposed an auto-associative memory model using an integrated short-term memory (STM) and long-term memory (LTM) spiking neural network. Their model requires that distinct pyramidal cells encoding different STM patterns are fired in different high-frequency gamma subcycles within each low-frequency theta oscillation. Auto-associative LTM is formed by modifying the recurrent synaptic efficacy between pyramidal cells. In order to store auto-associative LTM correctly, the recurrent synaptic efficacy must be bounded. The synaptic efficacy must be upper bounded to prevent re-firing of pyramidal cells in subsequent gamma subcycles. If cells encoding one memory item were to re-fire synchronously with other cells encoding another item in subsequent gamma subcycle, LTM stored via modifiable recurrent synapses would be corrupted. The synaptic efficacy must also be lower bounded so that memory pattern completion can be performed correctly. This paper uses the original model by Jensen et al. as the basis to illustrate the following points. Firstly, the importance of coordinated long-term memory (LTM) synaptic modification. Secondly, the use of a generic mathematical formulation (spiking response model) that can theoretically extend the results to other spiking network utilizing threshold-fire spiking neuron model. Thirdly, the interaction of long-term and short-term memory networks that possibly explains the asymmetric distribution of spike density in theta cycle through the merger of STM patterns with interaction of LTM network.     

Effect of channel block on the spiking activity of excitable membranes in a stochastic Hodgkin-Huxley model

The influence of intrinsic channel noise on the spontaneous spiking activity of poisoned excitable membrane patches is studied by use of a stochastic generalization of the Hodgkin-Huxley model. Internal noise stemming from the stochastic dynamics of individual ion channels is known to affect the collective properties of the whole ion channel cluster. For example, there exists an optimal size of the membrane patch for which the internal noise alone causes a regular spontaneous generation of action potentials. In addition to varying the size of ion channel clusters, living organisms may adapt the densities of ion channels in order to optimally regulate the spontaneous spiking activity. The influence of a channel block on the excitability of a membrane patch of a certain size is twofold: first, a variation of ion channel densities primarily yields a change of the conductance level; second, a down-regulation of working ion channels always increases the channel noise. While the former effect dominates in the case of sodium channel block resulting in a reduced spiking activity, the latter enhances the generation of spontaneous action potentials in the case of a tailored potassium channel blocking. Moreover, by blocking some portion of either potassium or sodium ion channels, it is possible to either increase or decrease the regularity of the spike train.     

Dynamics of networks of randomly connected excitatory and inhibitory spiking neurons

Recent advances in the understanding of the dynamics of populations of spiking neurones are reviewed. These studies shed light on how a population of neurones can follow arbitrary variations in input stimuli, how the dynamics of the population depends on the type of noise, and how recurrent connections influence the dynamics. The importance of inhibitory feedback for the generation of irregularity in single cell behaviour is emphasized. Examples of computation that recurrent networks with excitatory and inhibitory cells can perform are then discussed. Maintenance of a network state as an attractor of the system is discussed as a model for working memory function, in both object and spatial modalities. These models can be used to interpret and make predictions about electrophysiological data in the awake monkey.     

State-Dependent Effects of Na Channel Noise on Neuronal Burst Generation

We explore the effects of stochastic sodium (Na) channel activation on the variability and dynamics of spiking and bursting in a model neuron. The complete model segregates Hodgin-Huxley-type currents into two compartments, and undergoes applied current-dependent bifurcations between regimes of periodic bursting, chaotic bursting, and tonic spiking. Noise is added to simulate variable, finite sizes of the population of Na channels in the fast spiking compartment.During tonic firing, Na channel noise causes variability in interspike intervals (ISIs). The variance, as well as the sensitivity to noise, depend on the model's biophysical complexity. They are smallest in an isolated spiking compartment; increase significantly upon coupling to a passive compartment; and increase again when the second compartment also includes slow-acting currents. In this full model, sufficient noise can convert tonic firing into bursting.During bursting, the actions of Na channel noise are state-dependent. The higher the noise level, the greater the jitter in spike timing within bursts. The noise makes the burst durations of periodic regimes variable, while decreasing burst length duration and variance in a chaotic regime. Na channel noise blurs the sharp transitions of spike time and burst length seen at the bifurcations of the noise-free model. Close to such a bifurcation, the burst behaviors of previously periodic and chaotic regimes become essentially indistinguishable.We discuss biophysical mechanisms, dynamical interpretations and physiological implications. We suggest that noise associated with finite populations of Na channels could evoke very different effects on the intrinsic variability of spiking and bursting discharges, depending on a biological neuron's complexity and applied current-dependent state. We find that simulated channel noise in the model neuron qualitatively replicates the observed variability in burst length and interburst interval in an isolated biological bursting neuron.     

From sensors to spikes: evolving receptive fields to enhance sensorimotor information in a robot-arm

In biological systems, instead of actual encoders at different joints, proprioception signals are acquired through distributed receptive fields. In robotics, a single and accurate sensor output per link (encoder) is commonly used to track the position and the velocity. Interfacing bio-inspired control systems with spiking neural networks emulating the cerebellum with conventional robots is not a straight forward task. Therefore, it is necessary to adapt this one-dimensional measure (encoder output) into a multidimensional space (inputs for a spiking neural network) to connect, for instance, the spiking cerebellar architecture; i.e. a translation from an analog space into a distributed population coding in terms of spikes. This paper analyzes how evolved receptive fields (optimized towards information transmission) can efficiently generate a sensorimotor representation that facilitates its discrimination from other "sensorimotor states". This can be seen as an abstraction of the Cuneate Nucleus (CN) functionality in a robot-arm scenario. We model the CN as a spiking neuron population coding in time according to the response of mechanoreceptors during a multi-joint movement in a robot joint space. An encoding scheme that takes into account the relative spiking time of the signals propagating from peripheral nerve fibers to second-order somatosensory neurons is proposed. Due to the enormous number of possible encodings, we have applied an evolutionary algorithm to evolve the sensory receptive field representation from random to optimized encoding. Following the nature-inspired analogy, evolved configurations have shown to outperform simple hand-tuned configurations and other homogenized configurations based on the solution provided by the optimization engine (evolutionary algorithm). We have used artificial evolutionary engines as the optimization tool to circumvent nonlinearity responses in receptive fields.     

Intrinsic coherence resonance in excitable membrane patches

The influence of intrinsic channel noise on the spiking activity of excitable membrane patches is studied by use of a stochastic generalization of the Hodgkin-Huxley model. Internal noise stemming from the stochastic dynamics of individual ion channels does affect the electric properties of the cell-membrane patches. There exists an optimal size of the membrane patch for which the internal noise alone can cause a nearly regular spontaneous generation of action potentials. We consider the influence of intrinsic channel noise in presence of a constant and an oscillatory current driving for both, the mean interspike interval and the phenomenon of coherence resonance for neuronal spiking. Given small membrane patches, implying that channel noise dominates the excitable dynamics, we find the phenomenon of intrinsic coherence resonance. In this case, the relatively regular spiking behavior becomes essentially independent of an applied stimulus. We observed, however, the occurrence of a skipping of supra-threshold input events due to channel noise for intermediate patch sizes. This effect consequently reduces the overall coherence of the spiking.     

Running as fast as it can: How spiking dynamics form object groupings in the laminar circuits of visual cortex

How spiking neurons cooperate to control behavioral processes is a fundamental problem in computational neuroscience. Such cooperative dynamics are required during visual perception when spatially distributed image fragments are grouped into emergent boundary contours. Perceptual grouping is a challenge for spiking cells because its properties of collinear facilitation and analog sensitivity occur in response to binary spikes with irregular timing across many interacting cells. Some models have demonstrated spiking dynamics in recurrent laminar neocortical circuits, but not how perceptual grouping occurs. Other models have analyzed the fast speed of certain percepts in terms of a single feedforward sweep of activity, but cannot explain other percepts, such as illusory contours, wherein perceptual ambiguity can take hundreds of milliseconds to resolve by integrating multiple spikes over time. The current model reconciles fast feedforward with slower feedback processing, and binary spikes with analog network-level properties, in a laminar cortical network of spiking cells whose emergent properties quantitatively simulate parametric data from neurophysiological experiments, including the formation of illusory contours; the structure of non-classical visual receptive fields; and self-synchronizing gamma oscillations. These laminar dynamics shed new light on how the brain resolves local informational ambiguities through the use of properly designed nonlinear feedback spiking networks which run as fast as they can, given the amount of uncertainty in the data that they process.     

Anesthetics fragment hippocampal network activity,alter spine dynamics,and affect memory consolidation

General anesthesia is characterized by reversible loss of consciousness accompanied by transient amnesia. Yet, long-term memory impairment is an undesirable side effect. How different types of general anesthetics (GAs) affect the hippocampus, a brain region central to memory formation and consolidation, is poorly understood. Using extracellular recordings, chronic 2-photon imaging, and behavioral analysis, we monitor the effects of isoflurane (Iso), medetomidine/midazolam/fentanyl (MMF), and ketamine/xylazine (Keta/Xyl) on network activity and structural spine dynamics in the hippocampal CA1 area of adult mice. GAs robustly reduced spiking activity, decorrelated cellular ensembles, albeit with distinct activity signatures, and altered spine dynamics. CA1 network activity under all 3 anesthetics was different to natural sleep. Iso anesthesia most closely resembled unperturbed activity during wakefulness and sleep, and network alterations recovered more readily than with Keta/Xyl and MMF. Correspondingly, memory consolidation was impaired after exposure to Keta/Xyl and MMF, but not Iso. Thus, different anesthetics distinctly alter hippocampal network dynamics, synaptic connectivity, and memory consolidation, with implications for GA strategy appraisal in animal research and clinical settings.

Amnesia is a central part of the 200 million general anesthesia that are administered worldwide every year, but it is unclear how it affects the hippocampus. This mouse study explores how three commonly used anesthetics influence cellular network activity, spine dynamics and memory consolidation, finding that each of the three anesthetics alters the local field potential, spiking activity and cellular calcium dynamics in a unique way, and they all impact long-term spine dynamics.     

Analog VLSI implementation of resonate-and-fire neuron

We propose an analog integrated circuit that implements a resonate-and-fire neuron (RFN) model based on the Lotka-Volterra (LV) system. The RFN model is a spiking neuron model that has second-order membrane dynamics, and thus exhibits fast damped subthreshold oscillation, resulting in the coincidence detection, frequency preference, and post-inhibitory rebound. The RFN circuit has been derived from the LV system to mimic such dynamical behavior of the RFN model. Through circuit simulations, we demonstrate that the RFN circuit can act as a coincidence detector and a band-pass filter at circuit level even in the presence of additive white noise and background random activity. These results show that our circuit is expected to be useful for very large-scale integration (VLSI) implementation of functional spiking neural networks.     

Recurrent neural networks of integrate-and-fire cells simulating short-term memory and wrist movement tasks derived from continuous dynamic networks

Dynamic recurrent neural networks composed of units with continuous activation functions provide a powerful tool for simulating a wide range of behaviors, since the requisite interconnections can be readily derived by gradient descent methods. However, it is not clear whether more realistic integrate-and-fire cells with comparable connection weights would perform the same functions. We therefore investigated methods to convert dynamic recurrent neural networks of continuous units into networks with integrate-and-fire cells. The transforms were tested on two recurrent networks derived by backpropagation. The first simulates a short-term memory task with units that mimic neural activity observed in cortex of monkeys performing instructed delay tasks. The network utilizes recurrent connections to generate sustained activity that codes the remembered value of a transient cue. The second network simulates patterns of neural activity observed in monkeys performing a step-tracking task with flexion/extension wrist movements. This more complicated network provides a working model of the interactions between multiple spinal and supraspinal centers controlling motoneurons. Our conversion algorithm replaced each continuous unit with multiple integrate-and-fire cells that interact through delayed "synaptic potentials". Successful transformation depends on obtaining an appropriate fit between the activation function of the continuous units and the input-output relation of the spiking cells. This fit can be achieved by adapting the parameters of the synaptic potentials to replicate the input-output behavior of a standard sigmoidal activation function (shown for the short-term memory network). Alternatively, a customized activation function can be derived from the input-output relation of the spiking cells for a chosen set of parameters (demonstrated for the wrist flexion/extension network). In both cases the resulting networks of spiking cells exhibited activity that replicated the activity of corresponding continuous units. This confirms that the network solutions obtained through backpropagation apply to spiking networks and provides a useful method for deriving recurrent spiking networks performing a wide range of functions.     

In-phase and anti-phase synchronization in noisy Hodgkin–Huxley neurons

We numerically investigate the influence of intrinsic channel noise on the dynamical response of delay-coupling in neuronal systems. The stochastic dynamics of the spiking is modeled within a stochastic modification of the standard Hodgkin–Huxley model wherein the delay-coupling accounts for the finite propagation time of an action potential along the neuronal axon. We quantify this delay-coupling of the Pyragas-type in terms of the difference between corresponding presynaptic and postsynaptic membrane potentials. For an elementary neuronal network consisting of two coupled neurons we detect characteristic stochastic synchronization patterns which exhibit multiple phase-flip bifurcations: The phase-flip bifurcations occur in form of alternate transitions from an in-phase spiking activity towards an anti-phase spiking activity. Interestingly, these phase-flips remain robust for strong channel noise and in turn cause a striking stabilization of the spiking frequency.     

Neuromodulation by Glutamate and Acetylcholine can Change Circuit Dynamics by Regulating the Relative Influence of Afferent Input and Excitatory Feedback

Substances such as acetylcholine and glutamate act as both neurotransmitters and neuromodulators. As neuromodulators, they change neural information processing by regulating synaptic transmitter release, altering baseline membrane potential and spiking activity, and modifying long-term synaptic plasticity. Slice physiology research has demonstrated that many neuromodulators differentially modulate afferent, incoming information compared to intrinsic and recurrent processing in cortical structures such as piriform cortex, neocortex, and the hippocampus. The enhancement of afferent (external) pathways versus the suppression at recurrent (internal) pathways could cause cortical dynamics to switch between a predominant influence of external stimulation to a predominant influence of internal recall. Modulation of afferent versus intrinsic processing could contribute to the role of neuromodulators in regulating attention, learning, and memory effects in behavior.     

Dynamical coding of sensory information with competitive networks.

Based on experiments with the locust olfactory system, we demonstrate that model sensory neural networks with lateral inhibition can generate stimulus specific identity-temporal patterns in the form of stimulus-dependent switching among small and dynamically changing neural ensembles (each ensemble being a group of synchronized projection neurons). Networks produce this switching mode of dynamical activity when lateral inhibitory connections are strongly non-symmetric. Such coding uses 'winner-less competitive' (WLC) dynamics. In contrast to the well known winner-take-all competitive (WTA) networks and Hopfield nets, winner-less competition represents sensory information dynamically. Such dynamics are reproducible, robust against intrinsic noise and sensitive to changes in the sensory input. We demonstrate the validity of sensory coding with WLC networks using two different formulations of the dynamics, namely the average and spiking dynamics of projection neurons (PN).     

Learning Universal Computations with Spikes

Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g. for locomotion. Many such computations require previous building of intrinsic world models. Here we show how spiking neural networks may solve these different tasks. Firstly, we derive constraints under which classes of spiking neural networks lend themselves to substrates of powerful general purpose computing. The networks contain dendritic or synaptic nonlinearities and have a constrained connectivity. We then combine such networks with learning rules for outputs or recurrent connections. We show that this allows to learn even difficult benchmark tasks such as the self-sustained generation of desired low-dimensional chaotic dynamics or memory-dependent computations. Furthermore, we show how spiking networks can build models of external world systems and use the acquired knowledge to control them.     

Implications of different classes of sensorimotor disturbance for cerebellar-based motor learning models

The exact role of the cerebellum in motor control and learning is not yet fully understood. The structure, connectivity and plasticity within cerebellar cortex has been extensively studied, but the patterns of connectivity and interaction with other brain structures, and the computational significance of these patterns, is less well known and a matter of debate. Two contrasting models of the role of the cerebellum in motor adaptation have previously been proposed. Most commonly, the cerebellum is employed in a purely feedforward pathway, with its output contributing directly to the outgoing motor command. The cerebellum must then learn an inverse model of the motor apparatus in order to achieve accurate control. More recently, Porrill et al. (Proc Biol Sci 271(1541):789–796, 2004) and Porrill et al. (PLoS Comput Biol 3:1935–1950, 2007a) and Porrill et al. (Neural Comput 19(1), 170–193, 2007b) have highlighted the potential importance of these recurrent connections by proposing an alternative architecture in which the cerebellum is embedded in a recurrent loop with brainstem control circuitry. In this framework, the feedforward connections are not necessary at all. The cerebellum must learn a forward model of the motor apparatus for accurate motor commands to be generated. We show here how these two models exhibit contrasting yet complimentary learning capabilities. Central to the differences in performance between architectures is that there are two distinct kinds of disturbance to which a motor system may need to adapt (1) changes in the relationship between the motor command and the observed outcome and (2) changes in the relationship between the stimulus and the desired outcome. The computational distinction between these two kinds of transformation is subtle and has therefore often been overlooked. However, the implications for learning turn out to be significant: learning with a feedforward architecture is robust following changes in the stimulus-desired outcome mapping but not necessarily the motor command-outcome mapping, while learning with a recurrent architecture is robust under changes in the motor command-outcome mapping but not necessarily the stimulus-desired outcome mapping. We first analyse these differences theoretically and through simulations in the vestibulo-ocular reflex (VOR), then illustrate how these same concepts apply more generally with a model of reaching movements.     

A model of cerebellum stabilized and scheduled hybrid long-loop control of upright balance

A recurrent integrator proportional integral derivative (PID) model that has been used to account for cerebrocerebellar stabilization and scaling of transcortical proprioceptive feedback in the control of horizontal planar arm movements has been augmented with long-loop force~feedback and gainscheduling to describe the control of human upright balance. The cerebellar component of the controller is represented by two sets of gains that each provide linear scaling of same-joint and interjoint long-loop stretch responses between ankle, knee, and hip. The cerebral component of the model includes a single set of same-joint linear force feedback gains. Responses to platform translations of a three-segment body model operating under this hybrid proprioception and force-based long-loop control were simulated. With low-velocity platform disturbances, ankle-strategy-type postural recovery kinematics and electromyogram (EMG) patterns were generated using the first set of cerebeller control gains. With faster disturbances, balance was maintained by including the second set of gains cerebellar control gains that yielded mixed ankle-hip strategy-type kinematics and EMG patterns. The addition of small amounts of simulated muscular coactivation improved the fit to certain human datasets. It is proposed that the cerebellum switches control gainsets as a function of sensed body kinematic state. Reduction of cerebellar gains with a compensatory increase in muscular stiffness yielded posture recovery with abnormal motions consistent with those found in cerebellar disease. The model demonstrates that stabilized hybrid long-loop feedback with scheduling of linear gains may afford realistic balance control in the absence of explicit internal dynamics models and suggests that the cerebellum and cerebral cortex may contribute to balance control by such a mechanism.     

Dynamics of Competition between Subnetworks of Spiking Neuronal Networks in the Balanced State

We explore and analyze the nonlinear switching dynamics of neuronal networks with non-homogeneous connectivity. The general significance of such transient dynamics for brain function is unclear; however, for instance decision-making processes in perception and cognition have been implicated with it. The network under study here is comprised of three subnetworks of either excitatory or inhibitory leaky integrate-and-fire neurons, of which two are of the same type. The synaptic weights are arranged to establish and maintain a balance between excitation and inhibition in case of a constant external drive. Each subnetwork is randomly connected, where all neurons belonging to a particular population have the same in-degree and the same out-degree. Neurons in different subnetworks are also randomly connected with the same probability; however, depending on the type of the pre-synaptic neuron, the synaptic weight is scaled by a factor. We observed that for a certain range of the “within” versus “between” connection weights (bifurcation parameter), the network activation spontaneously switches between the two sub-networks of the same type. This kind of dynamics has been termed “winnerless competition”, which also has a random component here. In our model, this phenomenon is well described by a set of coupled stochastic differential equations of Lotka-Volterra type that imply a competition between the subnetworks. The associated mean-field model shows the same dynamical behavior as observed in simulations of large networks comprising thousands of spiking neurons. The deterministic phase portrait is characterized by two attractors and a saddle node, its stochastic component is essentially given by the multiplicative inherent noise of the system. We find that the dwell time distribution of the active states is exponential, indicating that the noise drives the system randomly from one attractor to the other. A similar model for a larger number of populations might suggest a general approach to study the dynamics of interacting populations of spiking networks.     

The effects of anatomical errors on shoulder kinematics computed using multi-body models

Joint motion calculated using multi-body models and inverse kinematics presents many advantages over direct marker-based calculations. However, the sensitivity of the computed kinematics is known to be partly caused by the model and could also be influenced by the participants’ anthropometry and sex. This study aimed to compare kinematics computed from an anatomical shoulder model based on medical images against a scaled-generic model and quantify the effects of anatomical errors and participants’ anthropometry on the calculated joint angles. Twelve participants have had planar shoulder movements experimentally captured in a motion lab, and their shoulder anatomy imaged using an MRI scanner. A shoulder multi-body dynamics model was developed for each participant, using both an image-based approach and a scaled-generic approach. Inverse kinematics have been performed using the two different modelling procedures and the three different experimental motions. Results have been compared using Bland–Altman analysis of agreement and further analysed using multi-linear regressions. Kinematics computed via an anatomical and a scaled-generic shoulder models differed in average from 3.2 to 5.4 degrees depending on the task. The MRI-based model presented smaller limits of agreement to direct kinematics than the scaled-generic model. Finally, the regression model predictors, including anatomical errors, sex, and BMI of the participant, explained from 41 to 80% of the kinematic variability between model types with respect to the task. This study highlighted the consequences of modelling precision, quantified the effects of anatomical errors on the shoulder kinematics, and showed that participants' anthropometry and sex could indirectly affect kinematic outcomes.