A Real-Time Brain-Machine Interface Combining Motor Target and Trajectory Intent Using an Optimal Feedback Control Design

Real-time brain-machine interfaces (BMI) have focused on either estimating the continuous movement trajectory or target intent. However, natural movement often incorporates both. Additionally, BMIs can be modeled as a feedback control system in which the subject modulates the neural activity to move the prosthetic device towards a desired target while receiving real-time sensory feedback of the state of the movement. We develop a novel real-time BMI using an optimal feedback control design that jointly estimates the movement target and trajectory of monkeys in two stages. First, the target is decoded from neural spiking activity before movement initiation. Second, the trajectory is decoded by combining the decoded target with the peri-movement spiking activity using an optimal feedback control design. This design exploits a recursive Bayesian decoder that uses an optimal feedback control model of the sensorimotor system to take into account the intended target location and the sensory feedback in its trajectory estimation from spiking activity. The real-time BMI processes the spiking activity directly using point process modeling. We implement the BMI in experiments consisting of an instructed-delay center-out task in which monkeys are presented with a target location on the screen during a delay period and then have to move a cursor to it without touching the incorrect targets. We show that the two-stage BMI performs more accurately than either stage alone. Correct target prediction can compensate for inaccurate trajectory estimation and vice versa. The optimal feedback control design also results in trajectories that are smoother and have lower estimation error. The two-stage decoder also performs better than linear regression approaches in offline cross-validation analyses. Our results demonstrate the advantage of a BMI design that jointly estimates the target and trajectory of movement and more closely mimics the sensorimotor control system.     

Tactile Gap Detection Deteriorates during Bimanual Symmetrical Movements under Mirror Visual Feedback

It has been suggested that incongruence between signals for motor intention and sensory input can cause pain and other sensory abnormalities. This claim is supported by reports that moving in an environment of induced sensorimotor conflict leads to elevated pain and sensory symptoms in those with certain painful conditions. Similar procedures can lead to reports of anomalous sensations in healthy volunteers too. In the present study, we used mirror visual feedback to investigate the effects of sensorimotor incongruence on responses to stimuli that arise from sources external to the body, in particular, touch. Incongruence between the sensory and motor signals for the right arm was manipulated by having the participants make symmetrical or asymmetrical movements while watching a reflection of their left arm in a parasagittal mirror, or the left hand surface of a similarly positioned opaque board. In contrast to our prediction, sensitivity to the presence of gaps in tactile stimulation of the right forearm was not reduced when participants made asymmetrical movements during mirror visual feedback, as compared to when they made symmetrical or asymmetrical movements with no visual feedback. Instead, sensitivity was reduced when participants made symmetrical movements during mirror visual feedback relative to the other three conditions. We suggest that small discrepancies between sensory and motor information, as they occur during mirror visual feedback with symmetrical movements, can impair tactile processing. In contrast, asymmetrical movements with mirror visual feedback may not impact tactile processing because the larger discrepancies between sensory and motor information may prevent the integration of these sources of information. These results contrast with previous reports of anomalous sensations during exposure to both low and high sensorimotor conflict, but are nevertheless in agreement with a forward model interpretation of perceptual modulations during goal directed movement.     

Grasping Hand Verbs: Oscillatory Beta and Alpha Correlates of Action-Word Processing

The grounded cognition framework proposes that sensorimotor brain areas, which are typically involved in perception and action, also play a role in linguistic processing. We assessed oscillatory modulation during visual presentation of single verbs and localized cortical motor regions by means of isometric contraction of hand and foot muscles. Analogously to oscillatory activation patterns accompanying voluntary movements, we expected a somatotopically distributed suppression of beta and alpha frequencies in the motor cortex during processing of body-related action verbs. Magnetoencephalographic data were collected during presentation of verbs that express actions performed using the hands (H) or feet (F). Verbs denoting no bodily movement (N) were used as a control. Between 150 and 500 msec after visual word onset, beta rhythms were suppressed in H and F in comparison with N in the left hemisphere. Similarly, alpha oscillations showed left-lateralized power suppression in the H-N contrast, although at a later stage. The cortical oscillatory activity that typically occurs during voluntary movements is therefore found to somatotopically accompany the processing of body-related verbs. The combination of a localizer task with the oscillatory investigation applied to verb reading as in the present study provides further methodological possibilities of tracking language processing in the brain.     

When Passive Feels Active - Delusion-Proneness Alters Self-Recognition in the Moving Rubber Hand Illusion

Psychotic patients have problems with bodily self-recognition such as the experience of self-produced actions (sense of agency) and the perception of the body as their own (sense of ownership). While it has been shown that such impairments in psychotic patients can be explained by hypersalient processing of external sensory input it has also been suggested that they lack normal efference copy in voluntary action. However, it is not known how problems with motor predictions like efference copy contribute to impaired sense of agency and ownership in psychosis or psychosis-related states. We used a rubber hand illusion based on finger movements and measured sense of agency and ownership to compute a bodily self-recognition score in delusion-proneness (indexed by Peters’ Delusion Inventory - PDI). A group of healthy subjects (n=71) experienced active movements (involving motor predictions) or passive movements (lacking motor predictions). We observed a highly significant correlation between delusion-proneness and self-recognition in the passive conditions, while no such effect was observed in the active conditions. This was seen for both ownership and agency scores. The result suggests that delusion-proneness is associated with hypersalient external input in passive conditions, resulting in an abnormal experience of the illusion. We hypothesize that this effect is not present in the active condition because deficient motor predictions counteract hypersalience in psychosis proneness.     

Altered processing of sensorimotor feedback in schizophrenia

Schizophrenia is characterized by an altered sense of the reality, associated with hallucinations and delusions. Some theories suggest that schizophrenia is related to a deficiency of the system that generates information about the sensory consequences of the actions realized by the subject. This system monitors the reafferent information resulting from an action and allows its anticipation. In the present study, we examined visual-event-related potentials (ERPs) generated by a sensorimotor task in 15 patients with schizophrenia and 15 normal controls. The visual feedback from hand movements performed by the subjects was experimentally distorted. Behavioral results showed that patients were impaired in recognizing their own movements. The ERP signal in patients also differed from those of control subjects. In patients, the ERP waveform was affected during the early part of the response (200 ms). This early effect in schizophrenic patients reveals a modified processing of the visual consequence of their actions.     

The neural basis of hallucinations and delusions

Schizophrenia is a biologically based disorder characterised by false perceptions (hallucinations) and false beliefs (delusions). The underlying physiological cause of these mental abnormalities remains unknown. There is increasing evidence that one class of symptom, the 'made experiences' including delusions of alien control and thought insertion, is associated with abnormalities in the mechanism that predicts the outcome of intended actions (the forward model). For these patients active movements feel like passive movements. As a result these patients do not feel in control of their actions. However, comparison with various neurological disorders, such as those associated with parietal lobe lesions, suggest that this abnormal experience is not sufficient to explain the feeling that some other agent is controlling is one's actions. Preliminary evidence suggests that patients with schizophrenia have an exaggerated sense of agency. In combination with the feeling of not being in control, this exaggerated sense of agency could explain delusions of alien control in which the patient attributes his own actions to another agent. Little is yet know about the neural basis of the predictive mechanisms that create the feeling that we are in control of our movements. Such prediction requires integration of information about intended movements generated in frontal cortex with sensory processing in posterior regions of the brain. Measures of functional connectivity suggest that long-range interactions between frontal and posterior regions are abnormally reduced in patients with schizophrenia. Further research is needed to explore the precise involvement of long-range connections in the mechanisms of forward modelling.     

Computational mechanisms of sensorimotor control

In order to generate skilled and efficient actions, the motor system must find solutions to several problems inherent in sensorimotor control, including nonlinearity, nonstationarity, delays, redundancy, uncertainty, and noise. We review these problems and five computational mechanisms that the brain may use to limit their deleterious effects: optimal feedback control, impedance control, predictive control, Bayesian decision theory, and sensorimotor learning. Together, these computational mechanisms allow skilled and fluent sensorimotor behavior.     

Monitoring and control of action by the frontal lobes

Success requires deciding among alternatives, controlling the initiation of movements, and judging the consequences of actions. When alternatives are difficult to distinguish, habitual responses must be overcome, or consequences are uncertain, deliberation is necessary and a supervisory system exerts control over the processes that produce sensory-guided movements. We have investigated these processes by recording neural activity in the frontal lobe of macaque monkeys performing a countermanding task. Distinct neurons in the frontal eye field respond to visual stimuli or control the production of the movements. In the supplementary eye field and anterior cingulate cortex, neurons appear not to control directly movement initiation but instead signal the production of errors, the anticipation and delivery of reinforcement, and the presence of processing conflict. These signals form the core of current models of supervisory control of sensorimotor processes.     

Modulation of brain activity during action observation: influence of perspective, transitivity and meaningfulness

The coupling process between observed and performed actions is thought to be performed by a fronto-parietal perception-action system including regions of the inferior frontal gyrus and the inferior parietal lobule. When investigating the influence of the movements' characteristics on this process, most research on action observation has focused on only one particular variable even though the type of movements we observe can vary on several levels. By manipulating the visual perspective, transitivity and meaningfulness of observed movements in a functional magnetic resonance imaging study we aimed at investigating how the type of movements and the visual perspective can modulate brain activity during action observation in healthy individuals. Importantly, we used an active observation task where participants had to subsequently execute or imagine the observed movements. Our results show that the fronto-parietal regions of the perception action system were mostly recruited during the observation of meaningless actions while visual perspective had little influence on the activity within the perception-action system. Simultaneous investigation of several sources of modulation during active action observation is probably an approach that could lead to a greater ecological comprehension of this important sensorimotor process.     

Infants in control: rapid anticipation of action outcomes in a gaze-contingent paradigm

Infants' poor motor abilities limit their interaction with their environment and render studying infant cognition notoriously difficult. Exceptions are eye movements, which reach high accuracy early, but generally do not allow manipulation of the physical environment. In this study, real-time eye tracking is used to put 6- and 8-month-old infants in direct control of their visual surroundings to study the fundamental problem of discovery of agency, i.e. the ability to infer that certain sensory events are caused by one's own actions. We demonstrate that infants quickly learn to perform eye movements to trigger the appearance of new stimuli and that they anticipate the consequences of their actions in as few as 3 trials. Our findings show that infants can rapidly discover new ways of controlling their environment. We suggest that gaze-contingent paradigms offer effective new ways for studying many aspects of infant learning and cognition in an interactive fashion and provide new opportunities for behavioral training and treatment in infants.     

Somatosensory-Motor Adaptation of Orofacial Actions in Posterior Parietal and Ventral Premotor Cortices

Recent studies have provided evidence for sensory-motor adaptive changes and action goal coding of visually guided manual action in premotor and posterior parietal cortices. To extend these results to orofacial actions, devoid of auditory and visual feedback, we used a repetition suppression paradigm while measuring neural activity with functional magnetic resonance imaging during repeated intransitive and silent lip, jaw and tongue movements. In the motor domain, this paradigm refers to decreased activity in specific neural populations due to repeated motor acts and has been proposed to reflect sensory-motor adaptation. Orofacial movements activated a set of largely overlapping, common brain areas forming a core neural network classically involved in orofacial motor control. Crucially, suppressed neural responses during repeated orofacial actions were specifically observed in the left ventral premotor cortex, the intraparietal sulcus, the inferior parietal lobule and the superior parietal lobule. Since no visual and auditory feedback were provided during orofacial actions, these results suggest somatosensory-motor adaptive control of intransitive and silent orofacial actions in these premotor and parietal regions.     

Visual attention and stability

In the present review, we address the relationship between attention and visual stability. Even though with each eye, head and body movement the retinal image changes dramatically, we perceive the world as stable and are able to perform visually guided actions. However, visual stability is not as complete as introspection would lead us to believe. We attend to only a few items at a time and stability is maintained only for those items. There appear to be two distinct mechanisms underlying visual stability. The first is a passive mechanism: the visual system assumes the world to be stable, unless there is a clear discrepancy between the pre- and post-saccadic image of the region surrounding the saccade target. This is related to the pre-saccadic shift of attention, which allows for an accurate preview of the saccade target. The second is an active mechanism: information about attended objects is remapped within retinotopic maps to compensate for eye movements. The locus of attention itself, which is also characterized by localized retinotopic activity, is remapped as well. We conclude that visual attention is crucial in our perception of a stable world.     

Both reaching and grasping are impacted by temporarily induced paresthesia

Abstract

Along with visual feedback, somatosensory feedback provides the nervous system with information regarding movement performance. Somatosensory system damage disrupts the normal feedback process, which can lead to a pins and needles sensation, or paresthaesia, and impaired movement control. The present study assessed the impact of temporarily induced median nerve paresthaesia, in individuals with otherwise intact sensorimotor function, on goal-directed reaching and grasping movements. Healthy, right-handed participants performed reach and grasp movements to five wooden Efron shapes, of which three were selected for analysis. Participants performed the task without online visual feedback and in two somatosensory conditions: 1) normal; and 2) disrupted somatosensory feedback. Disrupted somatosensory feedback was induced temporarily using a Digitimer (DS7AH) constant current stimulator. Participants’ movements to shapes 15 or 30?cm to the right of the hand’s start position were recorded using a 3?D motion analysis system at 300?Hz (Optotrak 3?D Investigator). Analyses revealed no significant differences for reaction time. Main effects for paresthaesia were observed for temporal and spatial aspects of the both the reach and grasp components of the movements. Although participants scaled their grip aperture to shape size under paresthaesia, the movements were smaller and more variable. Overall participants behaved as though they perceived they were performing larger and faster movements than they actually were. We suggest the presence of temporally induced paresthaesia affected online control by disrupting somatosensory feedback of the reach and grasp movements, ultimately leading to smaller forces and fewer corrective movements.     

Visual feedback is not necessary for the learning of novel dynamics

Background

When learning to perform a novel sensorimotor task, humans integrate multi-modal sensory feedback such as vision and proprioception in order to make the appropriate adjustments to successfully complete the task. Sensory feedback is used both during movement to control and correct the current movement, and to update the feed-forward motor command for subsequent movements. Previous work has shown that adaptation to stable dynamics is possible without visual feedback. However, it is not clear to what degree visual information during movement contributes to this learning or whether it is essential to the development of an internal model or impedance controller.

Methodology/Principle Findings

We examined the effects of the removal of visual feedback during movement on the learning of both stable and unstable dynamics in comparison with the case when both vision and proprioception are available. Subjects were able to learn to make smooth movements in both types of novel dynamics after learning with or without visual feedback. By examining the endpoint stiffness and force after learning it could be shown that subjects adapted to both types of dynamics in the same way whether they were provided with visual feedback of their trajectory or not. The main effects of visual feedback were to increase the success rate of movements, slightly straighten the path, and significantly reduce variability near the end of the movement.

Conclusions/Significance

These findings suggest that visual feedback of the hand during movement is not necessary for the adaptation to either stable or unstable novel dynamics. Instead vision appears to be used to fine-tune corrections of hand trajectory at the end of reaching movements.     

Kinematic and EMG characteristics of simple shoulder movements with proprioception and visual feedback.

The objective of this study was to determine if simple, shoulder movements use the dual control hypothesis strategy, previously demonstrated with elbow movements, and to see if this strategy also applies in the absence of visual feedback. Twenty subjects were seated with their right arm abducted to 90 degrees and externally rotated in the scapular plane. Subjects internally rotated to a target position using a custom shoulder wheel at three different speeds with and without visual feedback. Kinematics were collected with a motion analysis system and electromyographic (EMG) recordings of the pectoralis major (PECT), infraspinatus (INFRA), anterior and posterior (ADELT, PDELT) deltoid muscles were used to evaluate muscle activity patterns during movements. Kinematics changed as movement speed increased with less accuracy (p<0.01). Greater EMG activity was observed in the PECT, PDELT, and INFRA with shorter durations for the ADELT, PDELT and INFRA. Movements with only kinesthetic feedback were less accurate (p<0.01) and performed faster (p<0.01) than movements with visual feedback. EMG activity suggests no major difference in CNS control strategies in movements with and without visual feedback. Greater resolution with visual feedback enables the implementation of a dual control strategy, allowing greater movement velocity while maintaining accuracy.     

Neural Substrates for Judgment of Self-Agency in Ambiguous Situations

The sense of agency is the attribution of oneself as the cause of one’s own actions and their effects. Accurate agency judgments are essential for adaptive behaviors in dynamic environments, especially in conditions of uncertainty. However, it is unclear how agency judgments are made in ambiguous situations where self-agency and non-self-agency are both possible. Agency attribution is thus thought to require higher-order neurocognitive processes that integrate several possibilities. Furthermore, neural activity specific to self-attribution, as compared with non-self-attribution, may reflect higher-order critical operations that contribute to constructions of self-consciousness. Based on these assumptions, the present study focused on agency judgments under ambiguous conditions and examined the neural correlates of this operation with functional magnetic resonance imaging. Participants performed a simple but demanding agency-judgment task, which required them to report on whether they attributed their own action as the cause of a visual stimulus change. The temporal discrepancy between the participant’s action and the visual events was adaptively set to be maximally ambiguous for each individual on a trial-by-trial basis. Comparison with results for a control condition revealed that the judgment of agency was associated with activity in lateral temporo-parietal areas, medial frontal areas, the dorsolateral prefrontal area, and frontal operculum/insula regions. However, most of these areas did not differentiate between self- and non-self-attribution. Instead, self-attribution was associated with activity in posterior midline areas, including the precuneus and posterior cingulate cortex. These results suggest that deliberate self-attribution of an external event is principally associated with activity in posterior midline structures, which is imperative for self-consciousness.     

Decoding Hindlimb Movement for a Brain Machine Interface after a Complete Spinal Transection

Stereotypical locomotor movements can be made without input from the brain after a complete spinal transection. However, the restoration of functional gait requires descending modulation of spinal circuits to independently control the movement of each limb. To evaluate whether a brain-machine interface (BMI) could be used to regain conscious control over the hindlimb, rats were trained to press a pedal and the encoding of hindlimb movement was assessed using a BMI paradigm. Off-line, information encoded by neurons in the hindlimb sensorimotor cortex was assessed. Next neural population functions, or weighted representations of the neuronal activity, were used to replace the hindlimb movement as a trigger for reward in real-time (on-line decoding) in three conditions: while the animal could still press the pedal, after the pedal was removed and after a complete spinal transection. A novel representation of the motor program was learned when the animals used neural control to achieve water reward (e.g. more information was conveyed faster). After complete spinal transection, the ability of these neurons to convey information was reduced by more than 40%. However, this BMI representation was relearned over time despite a persistent reduction in the neuronal firing rate during the task. Therefore, neural control is a general feature of the motor cortex, not restricted to forelimb movements, and can be regained after spinal injury.     

Sensorimotor Control of Tracking Movements at Various Speeds for Stroke Patients as Well as Age-Matched and Young Healthy Subjects

There are aging- and stroke-induced changes on sensorimotor control in daily activities, but their mechanisms have not been well investigated. This study explored speed-, aging-, and stroke-induced changes on sensorimotor control. Eleven stroke patients (affected sides and unaffected sides) and 20 control subjects (10 young and 10 age-matched individuals) were enrolled to perform elbow tracking tasks using sinusoidal trajectories, which included 6 target speeds (15.7, 31.4, 47.1, 62.8, 78.5, and 94.2 deg/s). The actual elbow angle was recorded and displayed on a screen as visual feedback, and three indicators, the root mean square error (RMSE), normalized integrated jerk (NIJ) and integral of the power spectrum density of normalized speed (IPNS), were used to investigate the strategy of sensorimotor control. Both NIJ and IPNS had significant differences among the four groups (P<0.01), and the values were ranked in the following order: young controls < age-matched controls <unaffected sides of stroke patients <affected sides of stroke patients, which could be explained by the stroke- and aging-induced increase in reliance on feedback control. The RMSE increased with the increase in the target speed and the NIJ and IPNS initially declined and then remained steady for all four groups, which indicated a shift from feedback to feedforward control as the target speed increased. The feedback-feedforward trade-off induced by stroke, aging and speed might be explained by a change in the transmission delay and neuromotor noise. The findings in this study improve our understanding of the mechanism underlying the sensorimotor control and neurological changes caused by stroke and aging.     

An Electrocorticographic Brain Interface in an Individual with Tetraplegia

Brain-computer interface (BCI) technology aims to help individuals with disability to control assistive devices and reanimate paralyzed limbs. Our study investigated the feasibility of an electrocorticography (ECoG)-based BCI system in an individual with tetraplegia caused by C4 level spinal cord injury. ECoG signals were recorded with a high-density 32-electrode grid over the hand and arm area of the left sensorimotor cortex. The participant was able to voluntarily activate his sensorimotor cortex using attempted movements, with distinct cortical activity patterns for different segments of the upper limb. Using only brain activity, the participant achieved robust control of 3D cursor movement. The ECoG grid was explanted 28 days post-implantation with no adverse effect. This study demonstrates that ECoG signals recorded from the sensorimotor cortex can be used for real-time device control in paralyzed individuals.     

Adaptation to visual feedback delay influences visuomotor learning

Computational theory of motor control suggests that the brain continuously monitors motor commands, to predict their sensory consequences before actual sensory feedback becomes available. Such prediction error is a driving force of motor learning, and therefore appropriate associations between motor commands and delayed sensory feedback signals are crucial. Indeed, artificially introduced delays in visual feedback have been reported to degrade motor learning. However, considering our perceptual ability to causally bind our own actions with sensory feedback, demonstrated by the decrease in the perceived time delay following repeated exposure to an artificial delay, we hypothesized that such perceptual binding might alleviate deficits of motor learning associated with delayed visual feedback. Here, we evaluated this hypothesis by investigating the ability of human participants to adapt their reaching movements in response to a novel visuomotor environment with 3 visual feedback conditions--no-delay, sudden-delay, and adapted-delay. To introduce novelty into the trials, the cursor position, which originally indicated the hand position in baseline trials, was rotated around the starting position. In contrast to the no-delay condition, a 200-ms delay was artificially introduced between the cursor and hand positions during the presence of visual rotation (sudden-delay condition), or before the application of visual rotation (adapted-delay condition). We compared the learning rate (representing how the movement error modifies the movement direction in the subsequent trial) between the 3 conditions. In comparison with the no-delay condition, the learning rate was significantly degraded for the sudden-delay condition. However, this degradation was significantly alleviated by prior exposure to the delay (adapted-delay condition). Our data indicate the importance of appropriate temporal associations between motor commands and sensory feedback in visuomotor learning. Moreover, they suggest that the brain is able to account for such temporal associations in a flexible manner.