Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method

In this study, we have used a single link system with a pair of muscles that are excited with alpha and gamma signals to achieve both point to point and oscillatory movements with variable amplitude and frequency.

The system is highly nonlinear in all its physical and physiological attributes. The major physiological characteristics of this system are simultaneous activation of a pair of nonlinear muscle-like-actuators for control purposes, existence of nonlinear spindle-like sensors and Golgi tendon organ-like sensor, actions of gravity and external loading. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops.

A reinforcement learning method with an actor–critic (AC) architecture instead of middle and low level of central nervous system (CNS), is used to track a desired trajectory. The actor in this structure is a two layer feedforward neural network and the critic is a model of the cerebellum. The critic is trained by state-action-reward-state-action (SARSA) method. The critic will train the actor by supervisory learning based on the prior experiences. Simulation studies of oscillatory movements based on the proposed algorithm demonstrate excellent tracking capability and after 280 epochs the RMS error for position and velocity profiles were 0.02, 0.04?rad and rad/s, respectively.     

Rhythmic Movement of a Pair of One-Link Arms: Coordination by Intermittent Control

This paper considers the coordination and control of periodic movements of a pair of one-link arms. The system consists of two one-link arms each controlled by two muscle-like actuators. The muscle-like actuators are activated by simulated neural inputs. The model is simple enough to analyze, yet it embodies many aspects of human arms. Three attributes of the rhythmic coordinated movement of two arms, namely frequency, magnitude, and relative phase, are the only inputs to the controller. The controller uses mild co-activation and primarily activates the agonist. The effects of transmission delays, present in the reflex loop of physiological systems, also are modeled. The results of this research indicate the feasibility of controlling oscillatory body movements with short periods of activation. The result of many simulations, by varying the frequency or amplitude of the movement, indicate that the apparent lack of a simple relationship between neural control and desired behavior of the system should not be mistaken as evidence for the absence of a causal relationship between the activation patterns of the muscles and the desired behavior. Simulations of this system show stable oscillations at different frequencies and magnitudes even with additive noise and changes in the system mass.     

Rhythmic Movement of a Pair of One-Link Arms: Coordination by Intermittent Control

This paper considers the coordination and control of periodic movements of a pair of one-link arms. The system consists of two one-link arms each controlled by two muscle-like actuators. The muscle-like actuators are activated by simulated neural inputs. The model is simple enough to analyze, yet it embodies many aspects of human arms. Three attributes of the rhythmic coordinated movement of two arms, namely frequency, magnitude, and relative phase, are the only inputs to the controller. The controller uses mild co-activation and primarily activates the agonist. The effects of transmission delays, present in the reflex loop of physiological systems, also are modeled. The results of this research indicate the feasibility of controlling oscillatory body movements with short periods of activation. The result of many simulations, by varying the frequency or amplitude of the movement, indicate that the apparent lack of a simple relationship between neural control and desired behavior of the system should not be mistaken as evidence for the absence of a causal relationship between the activation patterns of the muscles and the desired behavior. Simulations of this system show stable oscillations at different frequencies and magnitudes even with additive noise and changes in the system mass.     

Physiological and pathological oscillatory networks in the human motor system.

Human brain functions are heavily contingent on neural interactions both at the single neuron and the neural population or system level. Accumulating evidence from neurophysiological studies strongly suggests that coupling of oscillatory neural activity provides an important mechanism to establish neural interactions. With the availability of whole-head magnetoencephalography (MEG) macroscopic oscillatory activity can be measured non-invasively from the human brain with high temporal and spatial resolution. To localise, quantify and map oscillatory activity and interactions onto individual brain anatomy we have developed the 'dynamic imaging of coherent sources' (DICS) method which allows to identify and analyse cerebral oscillatory networks from MEG recordings. Using this approach we have characterized physiological and pathological oscillatory networks in the human sensorimotor system. Coherent 8 Hz oscillations emerge from a cerebello-thalamo-premotor-motor cortical network and exert an 8 Hz oscillatory drive on the spinal motor neurons which can be observed as a physiological tremulousness of the movement termed movement discontinuities. This network represents the neurophysiological substrate of a discrete mode of motor control. In parkinsonian resting tremor we have identified an extensive cerebral network consisting of primary motor and lateral premotor cortex, supplementary motor cortex, thalamus/basal ganglia, posterior parietal cortex and secondary somatosensory cortex, which are entrained in the tremor or twice the tremor rhythm. This low frequency entrapment of motor areas likely plays an important role in the pathophysiology of parkinsonian motor symptoms. Finally, studies on patients with postural tremor in hepatic encephalopathy revealed that this type of tremor results from a pathologically slow thalamocortical and cortico-muscular coupling during isometric hold tasks. In conclusion, the analysis of oscillatory cerebral networks provides new insights into physiological mechanisms of motor control and pathophysiological mechanisms of tremor disorders.     

A hierarchical neural-network model for control and learning of voluntary movement

In order to control voluntary movements, the central nervous system (CNS) must solve the following three computational problems at different levels: the determination of a desired trajectory in the visual coordinates, the transformation of its coordinates to the body coordinates and the generation of motor command. Based on physiological knowledge and previous models, we propose a hierarchical neural network model which accounts for the generation of motor command. In our model the association cortex provides the motor cortex with the desired trajectory in the body coordinates, where the motor command is then calculated by means of long-loop sensory feedback. Within the spinocerebellum — magnocellular red nucleus system, an internal neural model of the dynamics of the musculoskeletal system is acquired with practice, because of the heterosynaptic plasticity, while monitoring the motor command and the results of movement. Internal feedback control with this dynamical model updates the motor command by predicting a possible error of movement. Within the cerebrocerebellum — parvocellular red nucleus system, an internal neural model of the inverse-dynamics of the musculo-skeletal system is acquired while monitoring the desired trajectory and the motor command. The inverse-dynamics model substitutes for other brain regions in the complex computation of the motor command. The dynamics and the inverse-dynamics models are realized by a parallel distributed neural network, which comprises many sub-systems computing various nonlinear transformations of input signals and a neuron with heterosynaptic plasticity (that is, changes of synaptic weights are assumed proportional to a product of two kinds of synaptic inputs). Control and learning performance of the model was investigated by computer simulation, in which a robotic manipulator was used as a controlled system, with the following results: (1) Both the dynamics and the inverse-dynamics models were acquired during control of movements. (2) As motor learning proceeded, the inverse-dynamics model gradually took the place of external feedback as the main controller. Concomitantly, overall control performance became much better. (3) Once the neural network model learned to control some movement, it could control quite different and faster movements. (4) The neural netowrk model worked well even when only very limited information about the fundamental dynamical structure of the controlled system was available. Consequently, the model not only accounts for the learning and control capability of the CNS, but also provides a promising parallel-distributed control scheme for a large-scale complex object whose dynamics are only partially known.     

Chaotic synchronization using a network of neural oscillators

Synchronization of chaotic low-dimensional systems has been a topic of much recent research. Such systems have found applications for secure communications. In this work we show how synchronization can be achieved in a high-dimensional chaotic neural network. The network used in our studies is an extension of the Hopfield Network, known as the Complex Hopfield Network (CHN). The CHN, also an associative memory, has both fixed point and limit cycle or oscillatory behavior. In the oscillatory mode, the network wanders chaotically from one stored pattern to another. We show how a pair of identical high-dimensional CHNs can be synchronized by communicating only a subset of state vector components. The synchronizability of such a system is characterized through simulations.     

Eye movement instabilities and nystagmus can be predicted by a nonlinear dynamics model of the saccadic system

The study of eye movements and oculomotor disorders has, for four decades, greatly benefitted from the application of control theoretic concepts. This paper is an example of a complementary approach based on the theory of nonlinear dynamical systems. Recently, a nonlinear dynamics model of the saccadic system was developed, comprising a symmetric piecewise-smooth system of six first-order autonomous ordinary differential equations. A preliminary numerical investigation of the model revealed that in addition to generating normal saccades, it could also simulate inaccurate saccades, and the oscillatory instability known as congenital nystagmus (CN). By varying the parameters of the model, several types of CN oscillations were produced, including jerk, bidirectional jerk and pendular nystagmus. The aim of this study was to investigate the bifurcations and attractors of the model, in order to obtain a classification of the simulated oculomotor behaviours. The application of standard stability analysis techniques, together with numerical work, revealed that the equations have a rich bifurcation structure. In addition to Hopf, homoclinic and saddlenode bifurcations organised by a Takens-Bogdanov point, the equations can undergo nonsmooth pitchfork bifurcations and nonsmooth gluing bifurcations. Evidence was also found for the existence of Hopf-initiated canards. The simulated jerk CN waveforms were found to correspond to a pair of post-canard symmetry-related limit cycles, which exist in regions of parameter space where the equations are a slow-fast system. The slow and fast phases of the simulated oscillations were attributed to the geometry of the corresponding slow manifold. The simulated bidirectional jerk and pendular waveforms were attributed to a symmetry invariant limit cycle produced by the gluing of the asymmetric cycles. In contrast to control models of the oculomotor system, the bifurcation analysis places clear restrictions on which kinds of behaviour are likely to be associated with each other in parameter space, enabling predictions to be made regarding the possible changes in the oscillation type that may be observed upon changing the model parameters. The analysis suggests that CN is one of a range of oculomotor disorders associated with a pathological saccadic braking signal, and that jerk and pendular nystagmus are the most probable oscillatory instabilities. Additionally, the transition from jerk CN to bidirectional jerk and pendular nystagmus observed experimentally when the gaze angle or attention level is changed is attributed to a gluing bifurcation. This suggests the possibility of manipulating the waveforms of subjects with jerk CN experimentally to produce waveforms with an extended foveation period, thereby improving visual resolution.     

Equilibrium point control of a monkey arm simulator by a fast learning tree structured artificial neural network

A planar 17 muscle model of the monkey's arm based on realistic biomechanical measurements was simulated on a Symbolics Lisp Machine. The simulator implements the equilibrium point hypothesis for the control of arm movements. Given initial and final desired positions, it generates a minimum-jerk desired trajectory of the hand and uses the backdriving algorithm to determine an appropriate sequence of motor commands to the muscles (Flash 1987; Mussa-Ivaldi et al. 1991; Dornay 1991b). These motor commands specify a temporal sequence of stable (attractive) equilibrium positions which lead to the desired hand movement. A strong disadvantage of the simulator is that it has no memory of previous computations. Determining the desired trajectory using the minimum-jerk model is instantaneous, but the laborious backdriving algorithm is slow, and can take up to one hour for some trajectories. The complexity of the required computations makes it a poor model for biological motor control. We propose a computationally simpler and more biologically plausible method for control which achieves the benefits of the backdriving algorithm. A fast learning, tree-structured network (Sanger 1991c) was trained to remember the knowledge obtained by the backdriving algorithm. The neural network learned the nonlinear mapping from a 2-dimensional cartesian planar hand position {x, y} to a 17-dimensional motor command space {u ₁, ..., u ₁₇}. Learning 20 training trajectories, each composed of 26 sample points {{x y{,{u ₁, ..., u ₁₇} took only 20 min on a Sun-4 Spare workstation. After the learning stage, new, untrained test trajectories as well as the original trajectories of the hand were given to the neural network as input. The network calculated the required motor commands for these movements. The resulting movements were close to the desired ones for both the training and test cases.     

Diurnal and ultradiurnal oscillations of growing organs within the framework of the information system of the plant

The analysis of growth and movements of seedling organs of kidney bean (Phaseolus vulgaris L.) provides a pattern of periodic phases of activity and relaxation. The existence of a central organ which would control the phase relationships, is not anticipated in the integrity of the plant. The cyclic activity of individual organs shows itself by growth associated with oscillation movements. One and the same organ may simultaneously accomplish oscillatory movements with a diurnal and ultradiurnal frequency. These rhythms originate during the organ development; the first pair of kidney bean leaves at first executes oscillation movements with a diurnal frequency and only after it is fully developed it exhibits a diurnal cycle with the photophil phase upwards and the scotophil downwards, the oscillations with an ultradiurnal oycle being maintained. The movements of the two leaves are synchronous, but there occur short sections with a desynchronous cycle. Simultaneously with these oscillations, in which the leaf petiole takes part, the adult leaf performs oscillatory movements perpendicular to the longitudinal leaf axis, the so-called side swings, controlled by periodical changes of the joint attaching the leaf blade. Their frequency is practically identical with that of the ultradiurnal cycle. Thus the periodic growth activity of the kidney bean results in growth oscillations passing in the diurnal cycle with a frequency of 0.043 rev.h^-1, their ascending and descending phases consisting of periodical ultradiurnal oscillations in cycles of 0.73–0.59 rev.h^-1. The epicotyl growth shows a similar pattern: into the basic diurnal nutation cycle with a frequency of 0.042 rev.h^-1 ultradiurnal oscillation cycles are incorporated having a similar frequency to that revealed in leaves (0.69–0.64 rev.h^-1). The diurnal oscillatory cycles belong to a system established on the basis of periodicity of day and night and other geophysical cycles. The ultradiurnal rhythmic oscillations are presumed to be an expression of the geocontrol system of root and shoot growth direction and orientation of the organ in space. The shape of their trajectories in bean leaves is contradictory to this; they are not spatial helices, as the kybernetic model would presuppose, but have a vertical, upwards and downwards course in one plane. Since these oscillatory movements with an ultradiurnal cycle cease after petiole excision from the stem and after shoot apex amputation, one may presume that they are coupled with the low-frequency oscillatory system of the epicotyl.     

A system identification analysis of neural adaptation dynamics and nonlinear responses in the local reflex control of locust hind limbs

Nonlinear type system identification models coupled with white noise stimulation provide an experimentally convenient and quick way to investigate the often complex and nonlinear interactions between the mechanical and neural elements of reflex limb control systems. Previous steady state analysis has allowed the neurons in such systems to be categorised by their sensitivity to position, velocity or acceleration (dynamics) and has improved our understanding of network function. These neurons, however, are known to adapt their output amplitude or spike firing rate during repetitive stimulation and this transient response may be more important than the steady state response for reflex control. In the current study previously used system identification methods are developed and applied to investigate both steady state and transient dynamic and nonlinear changes in the neural circuit responsible for controlling reflex movements of the locust hind limbs. Through the use of a parsimonious model structure and Monte Carlo simulations we conclude that key system dynamics remain relatively unchanged during repetitive stimulation while output amplitude adaptation is occurring. Whilst some evidence of a significant change was found in parts of the systems nonlinear response, the effect was small and probably of little physiological relevance. Analysis using biologically more realistic stimulation reinforces this conclusion.     

Cerebellar learning of accurate predictive control for fast-reaching movements

Long conduction delays in the nervous system prevent the accurate control of movements by feedback control alone. We present a new, biologically plausible cerebellar model to study how fast arm movements can be executed in spite of these delays. To provide a realistic test-bed of the cerebellar neural model, we embed the cerebellar network in a simulated biological motor system comprising a spinal cord model and a six-muscle two-dimensional arm model. We argue that if the trajectory errors are detected at the spinal cord level, memory traces in the cerebellum can solve the temporal mismatch problem between efferent motor commands and delayed error signals. Moreover, learning is made stable by the inclusion of the cerebello-nucleo-olivary loop in the model. It is shown that the cerebellar network implements a nonlinear predictive regulator by learning part of the inverse dynamics of the plant and spinal circuit. After learning, fast accurate reaching movements can be generated. Received: 8 February 1999 /Accepted in revised form: 7 August 1999     

Oscillatory neural networks in the rabbit hippocampus

A model is described to account for damped oscillatory activity of two interacting neural populations, pyramidal cells and interneurons. This network in the hippocampus is treated as a lumped system with tine delays between elements. The physiological mechanism underlying the oscillatory activity appears to involve neural population interaction and cannot be described in terms of a network composed of but two neurons, a single pyramidal cell and a single interneuron. An unusual aspect of the model is the explicit incorporation of an ongoing background input to raise the mean level of activity of the pyramidal cell population. This model has evolved from a series of studies previously performed on cats. To test the model experiments were performed on rabbits. The data showing oscillatory activity following fornix stimulation in the rabbit indicate that the model can be applied not only to the cat but also to the rabbit. In additions, for commissural stimulation oscillatory potentials of neural populations and individual pyramidal cells were evoked as predicted by the model.     

A neural network model of the vestibulo-ocular reflex using a local synaptic learning rule.

Vertebrates use the vestibulo-ocular reflex to maintain clear vision during head movements. This reflex requires eye-velocity commands from the semicircular canals to be integrated (mathematically) to produce eye-position commands for the extraocular muscles. This is accomplished by a neural network in the caudal pons. A model of this network is proposed using positive feedback via lateral inhibition. The model has been adapted to a learning network. We have developed a synaptic learning rule using only local information to make the model more physiological.     

动态神经网络中的同步振荡

目前有一种假设认为同一视觉对象是由一群神经元的同步振荡活动来表征的。这一神经元发放活动的时间特性,是解决视觉信息处理中“结合问题（Ｂｉｎｄｉｎｇｐｒｏｂｌｅｍ）”的可能机制。本文用我们所提出的一种简化现实性神经网络模型［１］所构造的时滞非线性振子网络［２］,模拟生物神经网络的同步振荡活动。并考虑了振子各参数的设置与振荡活动的关系,以及网络振子间耦联对同步活动的影响．     

Using strategic movement to calibrate a neural compass: a spiking network for tracking head direction in rats and robots

The head direction (HD) system in mammals contains neurons that fire to represent the direction the animal is facing in its environment. The ability of these cells to reliably track head direction even after the removal of external sensory cues implies that the HD system is calibrated to function effectively using just internal (proprioceptive and vestibular) inputs. Rat pups and other infant mammals display stereotypical warm-up movements prior to locomotion in novel environments, and similar warm-up movements are seen in adult mammals with certain brain lesion-induced motor impairments. In this study we propose that synaptic learning mechanisms, in conjunction with appropriate movement strategies based on warm-up movements, can calibrate the HD system so that it functions effectively even in darkness. To examine the link between physical embodiment and neural control, and to determine that the system is robust to real-world phenomena, we implemented the synaptic mechanisms in a spiking neural network and tested it on a mobile robot platform. Results show that the combination of the synaptic learning mechanisms and warm-up movements are able to reliably calibrate the HD system so that it accurately tracks real-world head direction, and that calibration breaks down in systematic ways if certain movements are omitted. This work confirms that targeted, embodied behaviour can be used to calibrate neural systems, demonstrates that 'grounding' of modelled biological processes in the real world can reveal underlying functional principles (supporting the importance of robotics to biology), and proposes a functional role for stereotypical behaviours seen in infant mammals and those animals with certain motor deficits. We conjecture that these calibration principles may extend to the calibration of other neural systems involved in motion tracking and the representation of space, such as grid cells in entorhinal cortex.     

On the activities in a continuous neural network

In this work an attempt is made to study the activities in a continuous neural system. The neural model considered here is a two dimensional continuous version of an earlier discrete model investigated in a series of papers [5–8]. The variations of the normalized firing rates in the present model are described by a nonlinear integro-partial differential equation. The conditions for the existence and uniqueness of the solutions of the describing equation subject to an initial condition are established and the steady-state solutions are investigated for inputs which are constant with respect to time. Depending on the parameters which are related to the self-inhibition and adaptation properties of the neural network, some of the oscillatory and stability properties of the solutions of the describing equation are discussed.     

Emergence of oscillations and spatio-temporal coherence states in a continuum-model of excitatory and inhibitory neurons

A neural field model of the reaction-diffusion type for the emergence of oscillatory phenomena in visual cortices is proposed. To investigate the joint spatio-temporal oscillatory dynamics in a continuous distribution of excitatory and inhibitory neurons, the coupling among oscillators is modelled as a diffusion process, combined with non-linear point interactions. The model exhibits cooperative activation properties in both time and space, by reacting to volleys of activations at multiple cortical sites with ordered spatio-temporal oscillatory states, similar to those found in the physiological experiments on slow-wave field potentials. The possible use of the resulting spatial distributions of coherent states, as a flexible medium to establish feature association, is discussed.     

鸣禽鸣叫学习的神经生物学机制

鸣禽鸣叫具有复杂的神经生理和生化基础,表现为一种复杂的学习过程。鸣啭控制系统是研究神经系统与学习、行为和发育关系的重要模型。而鸣禽鸣叫学习行为与鸣啭控制系统内长时程增强效应、神经元超微结构的改变和神经核团内的电活动、激素水平高低及其周期性变化、神经元再生或改变、即早基因的表达等方面密切相关。对鸣禽鸣叫的神经生物学机制进行了综述。     

A model cortical circuit for the storage of temporal sequences

Despite the fact that temporal information processing is of particular significance in biological memory systems, not much has yet been explored about how these systems manage to store temporal information involved in sequences of stimuli. A neural network model capable of learning and recalling temporal sequences is proposed, based on a neural mechanism in which the sequences are expanded into a series of periodic rectangular oscillations. Thus, the mathematical framework underlying the model, to some extent, is concerned with the Walsh function series. The oscillatory activities generated by the interplay between excitatory and inhibitory neuron pools are transmitted to another neuron pool whose role in learning and retrieval is to modify the rhythms and phases of the rectangular oscillations. Thus, a basic functional neural circuit involves three different neuron pools. The modifiability of rhythms and phases is incorporated into the model with the aim of improving the quality of the retrieval. Numerical simulations were conducted to show the characteristic features of the learning as well as the performance of the model in memory recall.     

Ventral striatum: a critical look at models of learning and evaluation

Extensive evidence implicates the ventral striatum in multiple distinct facets of action selection. Early work established a role in modulating ongoing behavior, as engaged by the energizing and directing influences of motivationally relevant cues and the willingness to expend effort in order to obtain reward. More recently, reinforcement learning models have suggested the notion of ventral striatum primarily as an evaluation step during learning, which serves as a critic to update a separate actor. Recent computational and experimental work may provide a resolution to the differences between these two theories through a careful parsing of behavior and the instrinsic heterogeneity that characterizes this complex structure.