Multimodal Movement Prediction - Towards an Individual Assistance of Patients

Assistive devices, like exoskeletons or orthoses, often make use of physiological data that allow the detection or prediction of movement onset. Movement onset can be detected at the executing site, the skeletal muscles, as by means of electromyography. Movement intention can be detected by the analysis of brain activity, recorded by, e.g., electroencephalography, or in the behavior of the subject by, e.g., eye movement analysis. These different approaches can be used depending on the kind of neuromuscular disorder, state of therapy or assistive device. In this work we conducted experiments with healthy subjects while performing self-initiated and self-paced arm movements. While other studies showed that multimodal signal analysis can improve the performance of predictions, we show that a sensible combination of electroencephalographic and electromyographic data can potentially improve the adaptability of assistive technical devices with respect to the individual demands of, e.g., early and late stages in rehabilitation therapy. In earlier stages for patients with weak muscle or motor related brain activity it is important to achieve high positive detection rates to support self-initiated movements. To detect most movement intentions from electroencephalographic or electromyographic data motivates a patient and can enhance her/his progress in rehabilitation. In a later stage for patients with stronger muscle or brain activity, reliable movement prediction is more important to encourage patients to behave more accurately and to invest more effort in the task. Further, the false detection rate needs to be reduced. We propose that both types of physiological data can be used in an and combination, where both signals must be detected to drive a movement. By this approach the behavior of the patient during later therapy can be controlled better and false positive detections, which can be very annoying for patients who are further advanced in rehabilitation, can be avoided.     

An Electrocorticographic Brain Interface in an Individual with Tetraplegia

Brain-computer interface (BCI) technology aims to help individuals with disability to control assistive devices and reanimate paralyzed limbs. Our study investigated the feasibility of an electrocorticography (ECoG)-based BCI system in an individual with tetraplegia caused by C4 level spinal cord injury. ECoG signals were recorded with a high-density 32-electrode grid over the hand and arm area of the left sensorimotor cortex. The participant was able to voluntarily activate his sensorimotor cortex using attempted movements, with distinct cortical activity patterns for different segments of the upper limb. Using only brain activity, the participant achieved robust control of 3D cursor movement. The ECoG grid was explanted 28 days post-implantation with no adverse effect. This study demonstrates that ECoG signals recorded from the sensorimotor cortex can be used for real-time device control in paralyzed individuals.     

Enhancing Sensorimotor Activity by Controlling Virtual Objects with Gaze

This fMRI work studies brain activity of healthy volunteers who manipulated a virtual object in the context of a digital game by applying two different control methods: using their right hand or using their gaze. The results show extended activations in sensorimotor areas, not only when participants played in the traditional way (using their hand) but also when they used their gaze to control the virtual object. Furthermore, with the exception of the primary motor cortex, regional motor activity was similar regardless of what the effector was: the arm or the eye. These results have a potential application in the field of the neurorehabilitation as a new approach to generate activation of the sensorimotor system to support the recovery of the motor functions.     

A Supplementary System for a Brain-Machine Interface Based on Jaw Artifacts for the Bidimensional Control of a Robotic Arm

Non-invasive Brain-Machine Interfaces (BMIs) are being used more and more these days to design systems focused on helping people with motor disabilities. Spontaneous BMIs translate user''s brain signals into commands to control devices. On these systems, by and large, 2 different mental tasks can be detected with enough accuracy. However, a large training time is required and the system needs to be adjusted on each session. This paper presents a supplementary system that employs BMI sensors, allowing the use of 2 systems (the BMI system and the supplementary system) with the same data acquisition device. This supplementary system is designed to control a robotic arm in two dimensions using electromyographical (EMG) signals extracted from the electroencephalographical (EEG) recordings. These signals are voluntarily produced by users clenching their jaws. EEG signals (with EMG contributions) were registered and analyzed to obtain the electrodes and the range of frequencies which provide the best classification results for 5 different clenching tasks. A training stage, based on the 2-dimensional control of a cursor, was designed and used by the volunteers to get used to this control. Afterwards, the control was extrapolated to a robotic arm in a 2-dimensional workspace. Although the training performed by volunteers requires 70 minutes, the final results suggest that in a shorter period of time (45 min), users should be able to control the robotic arm in 2 dimensions with their jaws. The designed system is compared with a similar 2-dimensional system based on spontaneous BMIs, and our system shows faster and more accurate performance. This is due to the nature of the control signals. Brain potentials are much more difficult to control than the electromyographical signals produced by jaw clenches. Additionally, the presented system also shows an improvement in the results compared with an electrooculographic system in a similar environment.     

Comparing information about arm movement direction in single channels of local and epicortical field potentials from monkey and human motor cortex.

Cortical field potentials have been used for decades in neurophysiological studies to probe spatio-temporal activity patterns of local populations of neurons. Recently, however, interest in these signals was spurred as they were proposed as potential control signals for neuronal motor prostheses, i.e., for devices fit to record and decode brain activity to restore motor functions in paralyzed patients. Little is known, however, about the functional significance of these cortical field potentials. Here we compared information about arm movement direction in two types of movement related cortical field potentials, obtained during a four direction center-out arm reaching paradigm: local field potentials (LFPs) recorded with intracortical micro-electrodes from monkey motor cortex, and epicortical field potentials (EFPs) recorded with macro-electrode arrays subdurally implanted on the surface of the human cerebral cortex. While monkey LFPs showed a typical sequence of positive and negative potential peaks, an initial negative peak was the most salient feature of human EFPs. Individual contralateral LFPs from the monkey motor cortex carried approximately twice as much decoded information (DI) about arm movement direction (median 0.27 bit) as did individual EFPs from the contralateral hand/arm area of primary motor cortex in humans (median 0.12 bit). This relation was similar to the relation between median peak signal-to-noise ratios for directional modulation of movement related potentials (MRPs) of both types of signals. We discuss possible reasons for the observed differences, amongst them epi- vs. intracortical recording and the different electrode dimensions used to measure EFPs and LFPs.     

Possible contributions of CPG activity to the control of rhythmic human arm movement

There is extensive modulation of cutaneous and H-reflexes during rhythmic leg movement in humans. Mechanisms controlling reflex modulation (e.g., phase- and task-dependent modulation, and reflex reversal) during leg movements have been ascribed to the activity of spinal central pattern generating (CPG) networks and peripheral feedback. Our working hypothesis has been that neural mechanisms (i.e., CPGs) controlling rhythmic movement are conserved between the human lumbar and cervical spinal cord. Thus reflex modulation during rhythmic arm movement should be similar to that for rhythmic leg movement. This hypothesis has been tested by studying the regulation of reflexes in arm muscles during rhythmic arm cycling and treadmill walking. This paper reviews recent studies that have revealed that reflexes in arm muscles show modulation within the movement cycle (e.g., phase-dependency and reflex reversal) and between static and rhythmic motor tasks (e.g., task-dependency). It is concluded that reflexes are modulated similarly during rhythmic movement of the upper and lower limbs, suggesting similar motor control mechanisms. One notable exception to this pattern is a failure of contralateral arm movement to modulate reflex amplitude, which contrasts directly with observations from the leg. Overall, the data support the hypothesis that CPG activity contributes to the neural control of rhythmic arm movement.     

Decoding Hindlimb Movement for a Brain Machine Interface after a Complete Spinal Transection

Stereotypical locomotor movements can be made without input from the brain after a complete spinal transection. However, the restoration of functional gait requires descending modulation of spinal circuits to independently control the movement of each limb. To evaluate whether a brain-machine interface (BMI) could be used to regain conscious control over the hindlimb, rats were trained to press a pedal and the encoding of hindlimb movement was assessed using a BMI paradigm. Off-line, information encoded by neurons in the hindlimb sensorimotor cortex was assessed. Next neural population functions, or weighted representations of the neuronal activity, were used to replace the hindlimb movement as a trigger for reward in real-time (on-line decoding) in three conditions: while the animal could still press the pedal, after the pedal was removed and after a complete spinal transection. A novel representation of the motor program was learned when the animals used neural control to achieve water reward (e.g. more information was conveyed faster). After complete spinal transection, the ability of these neurons to convey information was reduced by more than 40%. However, this BMI representation was relearned over time despite a persistent reduction in the neuronal firing rate during the task. Therefore, neural control is a general feature of the motor cortex, not restricted to forelimb movements, and can be regained after spinal injury.     

Redundant information encoding in primary motor cortex during natural and prosthetic motor control

Redundant encoding of information facilitates reliable distributed information processing. To explore this hypothesis in the motor system, we applied concepts from information theory to quantify the redundancy of movement-related information encoded in the macaque primary motor cortex (M1) during natural and neuroprosthetic control. Two macaque monkeys were trained to perform a delay center-out reaching task controlling a computer cursor under natural arm movement (manual control, ‘MC’), and using a brain-machine interface (BMI) via volitional control of neural ensemble activity (brain control, ‘BC’). During MC, we found neurons in contralateral M1 to contain higher and more redundant information about target direction than ipsilateral M1 neurons, consistent with the laterality of movement control. During BC, we found that the M1 neurons directly incorporated into the BMI (‘direct’ neurons) contained the highest and most redundant target information compared to neurons that were not incorporated into the BMI (‘indirect’ neurons). This effect was even more significant when comparing to M1 neurons of the opposite hemisphere. Interestingly, when we retrained the BMI to use ipsilateral M1 activity, we found that these neurons were more redundant and contained higher information than contralateral M1 neurons, even though ensembles from this hemisphere were previously less redundant during natural arm movement. These results indicate that ensembles most associated to movement contain highest redundancy and information encoding, which suggests a role for redundancy in proficient natural and prosthetic motor control.     

Spatial coding of visually guided arm movements in primate motor cortex

Previous studies of the motor cortex in behaving animals were focused on the relations between the activity of single cells, usually pyramidal tract neurons, and parameters of isometric contraction (e.g., intensity of force) or parameters of movement along one axis (e.g., flexion-extension) of a single joint (e.g., elbow or wrist). However, the commonly meaningful behavioral parameter is the trajectory of the hand in extrapersonal space, which is realized by simultaneous motions about two or three joints (e.g., elbow, shoulder, wrist) and concurrent engagement of several muscles. The spatial parameters of a straight trajectory are its direction and extent. We hypothesized that a major function of the motor cortex, among other possible roles, is the specification and control of the direction of the movement trajectory in space. This reference of motor cortical function to the control of spatial aspects of the trajectory differentiated our approach from the other approaches outlined above. We investigated the directional selectivity cells in the arm area of the motor cortex by recording their activity while monkeys moved their hands in various directions in space towards visual targets. There were two salient findings of these studies. First, the intensity of the discharge of single cells varies in an orderly fashion with the direction of movement in space, so that the discharge rate is highest with movements in a preferred direction, and decreases progressively with movements made in directions more and more away from the preferred one. Thus single cells are broadly tuned around a preferred direction which differs among different cells.(ABSTRACT TRUNCATED AT 250 WORDS)     

Using Reinforcement Learning to Provide Stable Brain-Machine Interface Control Despite Neural Input Reorganization

Brain-machine interface (BMI) systems give users direct neural control of robotic, communication, or functional electrical stimulation systems. As BMI systems begin transitioning from laboratory settings into activities of daily living, an important goal is to develop neural decoding algorithms that can be calibrated with a minimal burden on the user, provide stable control for long periods of time, and can be responsive to fluctuations in the decoder’s neural input space (e.g. neurons appearing or being lost amongst electrode recordings). These are significant challenges for static neural decoding algorithms that assume stationary input/output relationships. Here we use an actor-critic reinforcement learning architecture to provide an adaptive BMI controller that can successfully adapt to dramatic neural reorganizations, can maintain its performance over long time periods, and which does not require the user to produce specific kinetic or kinematic activities to calibrate the BMI. Two marmoset monkeys used the Reinforcement Learning BMI (RLBMI) to successfully control a robotic arm during a two-target reaching task. The RLBMI was initialized using random initial conditions, and it quickly learned to control the robot from brain states using only a binary evaluative feedback regarding whether previously chosen robot actions were good or bad. The RLBMI was able to maintain control over the system throughout sessions spanning multiple weeks. Furthermore, the RLBMI was able to quickly adapt and maintain control of the robot despite dramatic perturbations to the neural inputs, including a series of tests in which the neuron input space was deliberately halved or doubled.     

Anticipatory postural adjustments to arm movement reveal complex control of paraspinal muscles in the thorax

Anatomical and empirical data suggest that deep and superficial muscles may have different functions for thoracic spine control. This study investigated thoracic paraspinal muscle activity during anticipatory postural adjustments associated with arm movement. Electromyographic (EMG) recordings were made from the right deep (multifidus/rotatores) and superficial (longissimus) muscles at T5, T8, and T11 levels using fine-wire electrodes. Ten healthy participants performed fast unilateral and bilateral flexion and extension arm movements in response to a light. EMG amplitude was measured during 25 ms epochs for 150 ms before and 400 ms after deltoid EMG onset. During arm flexion movements, multifidus and longissimus had two bursts of activity, one burst prior to deltoid and a late burst. With arm extension both muscles were active in a single burst after deltoid onset. There was differential activity with respect to direction of trunk rotation induced by arm movement. Right longissimus was most active with left arm movements and right multifidus was most active with right arm movements. All levels of the thorax responded similarly. We suggest that although thoracic multifidus and longissimus function similarly to control sagittal plane perturbations, these muscles are differentially active with rotational forces on the trunk.     

Simulating Discrete and Rhythmic Multi-joint Human Arm Movements by Optimization of Nonlinear Performance Indices

An optimization approach applied to mechanical linkage models is used to simulate human arm movements. Predicted arm trajectories are the result of minimizing a nonlinear performance index that depends on kinematic or dynamic variables of the movement. A robust optimization algorithm is presented that computes trajectories which satisfy the necessary conditions with high accuracy. It is especially adapted to the analysis of discrete and rhythmic movements. The optimization problem is solved by parameterizing each generalized coordinate (e.g., joint angular displacement) in terms of Jacobi polynomials and Fourier series, depending on whether discrete or rhythmic movements are considered, combined with a multiple shooting algorithm. The parameterization of coordinates has two advantages. First, it provides an initial guess for the multiple shooting algorithm which solves the optimization problem with high accuracy. Second, it leads to a low dimensional representation of discrete and rhythmic movements in terms of expansion coefficients. The selection of a suitable feature space is an important prerequisite for comparison, recognition and classification of movements. In addition, the separate computational analysis of discrete and rhythmic movements is motivated by their distinct neurophysiological realizations in the cortex. By investigating different performance indices subject to different boundary conditions, the approach can be used to examine possible strategies that humans adopt in selecting specific arm motions for the performance of different tasks in a plane and in three-dimensional space.     

A role for the cerebellum in the control of limb movement velocity

Accepting, rejecting or modifying the many different theories of the cerebellum's role in the control of movement requires an understanding of the signals encoded in the discharge of cerebellar neurons and how those signals are transformed by the cerebellar circuitry. Particularly challenging is understanding the sensory and motor signals carried by the two types of action potentials generated by cerebellar Purkinje cells, the simple spikes and complex spikes. Advances have been made in understanding this signal processing in the context of voluntary arm movements. Recent evidence suggests that mossy fiber afferents to the cerebellar cortex are a source of kinematic signals, providing information about movement direction and speed. In turn, the simple spike discharge of Purkinje cells integrates this mossy fiber information to generate a movement velocity signal. Complex spikes may signal errors in movement velocity. It is proposed that the cerebellum uses the signals carried by the simple and complex spike discharges to control movement velocity for both step and tracking arm movements.     

Real-Time Subject-Independent Pattern Classification of Overt and Covert Movements from fNIRS Signals

Recently, studies have reported the use of Near Infrared Spectroscopy (NIRS) for developing Brain–Computer Interface (BCI) by applying online pattern classification of brain states from subject-specific fNIRS signals. The purpose of the present study was to develop and test a real-time method for subject-specific and subject-independent classification of multi-channel fNIRS signals using support-vector machines (SVM), so as to determine its feasibility as an online neurofeedback system. Towards this goal, we used left versus right hand movement execution and movement imagery as study paradigms in a series of experiments. In the first two experiments, activations in the motor cortex during movement execution and movement imagery were used to develop subject-dependent models that obtained high classification accuracies thereby indicating the robustness of our classification method. In the third experiment, a generalized classifier-model was developed from the first two experimental data, which was then applied for subject-independent neurofeedback training. Application of this method in new participants showed mean classification accuracy of 63% for movement imagery tasks and 80% for movement execution tasks. These results, and their corresponding offline analysis reported in this study demonstrate that SVM based real-time subject-independent classification of fNIRS signals is feasible. This method has important applications in the field of hemodynamic BCIs, and neuro-rehabilitation where patients can be trained to learn spatio-temporal patterns of healthy brain activity.     

Are interlimb interactions disturbed in patients with Parkinson’s disease? A study under unloading conditions

During natural human locomotion, neural connections are activated that are typical of regulation of the quadrupedal walking. The interaction between the neural networks generating rhythmic movements of the upper and lower limbs depends on tonic state of each of these networks regulated by motor signals from the brain. Distortion of these signals in patients with Parkinson’s disease (PD) may lead to disruption of the interlimb interactions. We examined the effect of movements of the limbs of one girdle on the parameters of the motor activity of another limb girdle at their joint cyclic movements under the conditions of arm and leg unloading in 17 patients with PD and 16 healthy subjects. We have shown that, in patients, the effect of voluntary and passive movements of arms, as well as the active movement of the distal parts of arms, on the voluntary movement of legs is weak, while in healthy subjects, the effect of arm movements on the parameters of voluntary stepping is significant. The effect of arm movements on the activation of the involuntary stepping by vibrational stimulation of-legs in patients was absent, while in healthy subjects, the motor activity of arms increased the possibility of involuntary rhythmic movements activation. Differences in the effect of leg movements on the rhythmic movements of arms were found in both patients and healthy subjects. The interlimb interaction appeared after drug administration. However, the effect of the drug was not sufficient for the recovery of normal state of the neural networks in patients. In PD patients, neural networks generating stepping rhythm have an increased tonic activity, which prevents the activation and appearance of involuntary rhythmic movements facilitating the effects of arms on legs.     

The discontinuous nature of motor execution

Human movement control requires adequate coordination of different movements, which is particularly important when different motor tasks are simultaneously executed by the same effector(s) (e.g. a muscle or a joint). The process of movement execution involves a series of highly nonlinear elements; for instance, a motor unit of a muscle produces force only in the direction of muscle shortening, thus representing a threshold operator that transforms the bipolar (i.e. excitatory or inhibitory) information at its spinal input into a purely unipolar signal (i.e. muscle force). This tripartite research report addresses the contribution of the nonlinearity of neuromuscular elements to the coordination of different motor tasks simultaneously executed by the same limb. In this first part of the series, a new hypothesis for such a single-muscle multiple-task coordination is presented which suggests an essentially threshold-linear coordination mechanism. Control signals generated by the central nervous system for each individual movement independently and feedback information from peripheral receptors are linearly superimposed. This compound control/feedback signal is processed by a nonlinear limiter element reflecting the discontinuous properties of the muscle and its reflex circuitry. It is shown that threshold-linear interaction of descending commands and afferent feedback information can lead to complex interdependent patterns of compound motor action. This includes the possibility of gating (i.e. the ability of one movement pattern to constrain or even impede the execution of another pattern) and of delayed response initiation when simultaneously performing more than one voluntary motor task. A theoretical analysis of the threshold-linear coordination mechanism and an extensive experimental validation of the model is provided in part II and part III of the report. Received: 6 October 1998 / Accepted in revised form: 2 June 1999     

Grasp Detection from Human ECoG during Natural Reach-to-Grasp Movements

Various movement parameters of grasping movements, like velocity or type of the grasp, have been successfully decoded from neural activity. However, the question of movement event detection from brain activity, that is, decoding the time at which an event occurred (e.g. movement onset), has been addressed less often. Yet, this may be a topic of key importance, as a brain-machine interface (BMI) that controls a grasping prosthesis could be realized by detecting the time of grasp, together with an optional decoding of which type of grasp to apply. We, therefore, studied the detection of time of grasps from human ECoG recordings during a sequence of natural and continuous reach-to-grasp movements. Using signals recorded from the motor cortex, a detector based on regularized linear discriminant analysis was able to retrieve the time-point of grasp with high reliability and only few false detections. Best performance was achieved using a combination of signal components from time and frequency domains. Sensitivity, measured by the amount of correct detections, and specificity, represented by the amount of false detections, depended strongly on the imposed restrictions on temporal precision of detection and on the delay between event detection and the time the event occurred. Including neural data from after the event into the decoding analysis, slightly increased accuracy, however, reasonable performance could also be obtained when grasping events were detected 125 ms in advance. In summary, our results provide a good basis for using detection of grasping movements from ECoG to control a grasping prosthesis.     

Coherent activity between auditory and visual modalities during the induction of peacefulness

Multisensory integration involves combining information from different senses to create a perception. The diverse characteristics of different sensory systems make it interesting to determine how cooperation and competition contribute to emotional experiences. Therefore, the aim of this study were to estimate the bias from the match attributes of the auditory and visual modalities and to depict specific brain activity frequency (theta, alpha, beta, and gamma) patterns related to a peaceful mood by using magnetoencephalography. The present study provides evidence of auditory domination in perceptual bias during multimodality processing of peaceful consciousness. Coherence analysis suggested that the theta oscillations are a transmitter of emotion signals, with the left and right brains being active in peaceful and fearful moods, respectively. Notably, hemispheric lateralization was also apparent in the alpha and beta oscillations, which might govern simple or pure information (e.g. from single modality) in the right brain but complex or mixed information (e.g. from multiple modalities) in the left brain.     

Optical mapping system with real-time control capability

Real-time, closed-loop intervention is an emerging experiment-control method that promises to provide invaluable new insight into cardiac electrophysiology. One example is the investigation of closed-loop feedback control of cardiac activity (e.g., alternans) as a possible method of preventing arrhythmia onset. To date, such methods have been investigated only in vitro using microelectrode systems, which are hindered by poor spatial resolution and are not well suited for atrial or ventricular tissue preparations. We have developed a system that uses optical mapping techniques and an electrical stimulator as the sensory and effector arms, respectively, of a closed-loop, real-time control system. The system consists of a 2,048 x 1 pixel line-scan charge-coupled device camera that records optical signals from the tissue. Custom-image processing and control software, which is implemented on top of a hard real-time operation system (RTAI Linux), process the data and make control decisions with a deterministic delay of <1 ms. The system is tested in two ways: 1) it is used to control, in real time, simulated optical signals of electrical alternans; and 2) it uses precisely timed, feedback-controlled initiation of antitachycardia pacing to terminate reentrant arrhythmias in an arterially perfused swine right ventricle stained with voltage-sensitive fluorescent dye 4{beta-[2-(di-n-butylamino)-6-napathy]vinyl}pyridinium (di-4-ANEPPS). Thus real-time control of cardiac activity using optical mapping techniques is feasible. Such a system is attractive because it offers greater measurement resolution than the electrode-based systems with which real-time control has been used previously.     

Independent Mobility Achieved through a Wireless Brain-Machine Interface

Individuals with tetraplegia lack independent mobility, making them highly dependent on others to move from one place to another. Here, we describe how two macaques were able to use a wireless integrated system to control a robotic platform, over which they were sitting, to achieve independent mobility using the neuronal activity in their motor cortices. The activity of populations of single neurons was recorded using multiple electrode arrays implanted in the arm region of primary motor cortex, and decoded to achieve brain control of the platform. We found that free-running brain control of the platform (which was not equipped with any machine intelligence) was fast and accurate, resembling the performance achieved using joystick control. The decoding algorithms can be trained in the absence of joystick movements, as would be required for use by tetraplegic individuals, demonstrating that the non-human primate model is a good pre-clinical model for developing such a cortically-controlled movement prosthetic. Interestingly, we found that the response properties of some neurons differed greatly depending on the mode of control (joystick or brain control), suggesting different roles for these neurons in encoding movement intention and movement execution. These results demonstrate that independent mobility can be achieved without first training on prescribed motor movements, opening the door for the implementation of this technology in persons with tetraplegia.