Encoding brain network response to free viewing of videos

A challenging goal for cognitive neuroscience researchers is to determine how mental representations are mapped onto the patterns of neural activity. To address this problem, functional magnetic resonance imaging (fMRI) researchers have developed a large number of encoding and decoding methods. However, previous studies typically used rather limited stimuli representation, like semantic labels and Wavelet Gabor filters, and largely focused on voxel-based brain patterns. Here, we present a new fMRI encoding model to predict the human brain’s responses to free viewing of video clips which aims to deal with this limitation. In this model, we represent the stimuli using a variety of representative visual features in the computer vision community, which can describe the global color distribution, local shape and spatial information and motion information contained in videos, and apply the functional connectivity to model the brain’s activity pattern evoked by these video clips. Our experimental results demonstrate that brain network responses during free viewing of videos can be robustly and accurately predicted across subjects by using visual features. Our study suggests the feasibility of exploring cognitive neuroscience studies by computational image/video analysis and provides a novel concept of using the brain encoding as a test-bed for evaluating visual feature extraction.     

Rhesus monkeys see who they hear: spontaneous cross-modal memory for familiar conspecifics

Rhesus monkeys gather much of their knowledge of the social world through visual input and may preferentially represent this knowledge in the visual modality. Recognition of familiar faces is clearly advantageous, and the flexibility and utility of primate social memory would be greatly enhanced if visual memories could be accessed cross-modally either by visual or auditory stimulation. Such cross-modal access to visual memory would facilitate flexible retrieval of the knowledge necessary for adaptive social behavior. We tested whether rhesus monkeys have cross-modal access to visual memory for familiar conspecifics using a delayed matching-to-sample procedure. Monkeys learned visual matching of video clips of familiar individuals to photographs of those individuals, and generalized performance to novel videos. In crossmodal probe trials, coo-calls were played during the memory interval. The calls were either from the monkey just seen in the sample video clip or from a different familiar monkey. Even though the monkeys were trained exclusively in visual matching, the calls influenced choice by causing an increase in the proportion of errors to the picture of the monkey whose voice was heard on incongruent trials. This result demonstrates spontaneous cross-modal recognition. It also shows that viewing videos of familiar monkeys activates naturally formed memories of real monkeys, validating the use of video stimuli in studies of social cognition in monkeys.     

Modifications of EEG Activity Related to Perception of Emotionally Colored,Erotic, and Neutral Pictures in Women during Different Phases of the Ovulatory (Menstrual) Cycle

We studied modifications of EEG activity related to perception of visual stimuli (pictures) of the International Affective Picture System (IAPS) in women during different phases of the ovulatory (menstrual) cycle. We found that the woman’s brain is most intensely activated by both emotionally negative and positive visual stimuli during the follicular phase of the ovulatory cycle, while such activation is minimum during ovulation per se. Upon the action of neutral stimuli, cerebral activation was the most intense during the lutein phase (compared with that within other phases); manifestations of activation were concentrated in the right hemisphere. Cognitive and emotional components of perception of affective pictures were expressed to a greatest extent in the course of viewing positive stimuli during the follicular phase. Perception of emotionally pleasant stimuli during other phases was accompanied by lateralization of activation of sensory and analytical processes in the left hemisphere during the ovulatory phase and in the right hemisphere within the lutein phase. The viewing of emotionally negative pictures during the follicular phase led to a rise in the power of theta oscillations in the left frontal region and also to depression of the alpha activity in central/parietal parts of the left hemisphere, which can result from aggravation of anxiety and verbally shaped disturbing ideas upon the action of such stimuli. Presentation of erotically colored visual stimuli caused the most intense changes in the EEG, which depended on the phases of the ovulatory cycle, during post-stimulation time interval but not during the viewing of pictures itself.     

Mirroring Pain in the Brain: Emotional Expression versus Motor Imitation

Perception of pain in others via facial expressions has been shown to involve brain areas responsive to self-pain, biological motion, as well as both performed and observed motor actions. Here, we investigated the involvement of these different regions during emotional and motor mirroring of pain expressions using a two-task paradigm, and including both observation and execution of the expressions. BOLD responses were measured as subjects watched video clips showing different intensities of pain expression and, after a variable delay, either expressed the amount of pain they perceived in the clips (pain task), or imitated the facial movements (movement task). In the pain task condition, pain coding involved overlapping activation across observation and execution in the anterior cingulate cortex, supplementary motor area, inferior frontal gyrus/anterior insula, and the inferior parietal lobule, and a pain-related increase (pain vs. neutral) in the anterior cingulate cortex/supplementary motor area, the right inferior frontal gyrus, and the postcentral gyrus. The ‘mirroring’ response was stronger in the inferior frontal gyrus and middle temporal gyrus/superior temporal sulcus during the pain task, and stronger in the inferior parietal lobule in the movement task. These results strongly suggest that while motor mirroring may contribute to the perception of pain expressions in others, interpreting these expressions in terms of pain content draws more heavily on networks involved in the perception of affective meaning.     

Impairment of Emotional Memory and Ability to Identify Emotional States in Patients with Parkinson's Disease

Parkinson's disease (PD) is characterized not only by a progressive movement disorder but also by a variety of mental disorders, which can be arbitrarily divided into cognitive and emotional–personality disorders. The aim of this investigation was to study the relationship between cognitive and emotional–personality disturbances in patients with PD during learning of emotionally significant words and visual recognition of emotional states. The volume and stability of recall of neutral and expressive words were significantly lower in patients than in healthy subjects, whereas the parameters of word recognition were similar in both groups. Unlike in healthy subjects, the characteristics of recall of neutral and expressive words in patients did not differ considerably. Patients with PD significantly more often incorrectly interpreted emotional states shown in pictures, particularly negative emotions. Deficit in retention of expressive words and recognition of emotional states correlated with the severity of motor and cognitive disorders, particularly of attention and executive dysfunction. The results of our study can be explained by a dissociation of the cognitive and motivational–affective functions in PD because of a disconnection between the basal ganglia, the limbic structures, and the frontal cortex. This dissociation tended to increase with progress of the disease.     

Decoding visual roughness perception: an fMRI study

Abstract

The neural substrates of tactile roughness perception have been investigated by many neuroimaging studies, while relatively little effort has been devoted to the investigation of neural representations of visually perceived roughness. In this human fMRI study, we looked for neural activity patterns that could be attributed to five different roughness intensity levels when the stimuli were perceived visually, i.e., in absence of any tactile sensation. During functional image acquisition, participants viewed video clips displaying a right index fingertip actively exploring the sandpapers that had been used for the behavioural experiment. A whole brain multivariate pattern analysis found four brain regions in which visual roughness intensities could be decoded: the bilateral posterior parietal cortex (PPC), the primary somatosensory cortex (S1) extending to the primary motor cortex (M1) in the right hemisphere, and the inferior occipital gyrus (IOG). In a follow-up analysis, we tested for correlations between the decoding accuracies and the tactile roughness discriminability obtained from a preceding behavioural experiment. We could not find any correlation between both although, during scanning, participants were asked to recall the tactilely perceived roughness of the sandpapers. We presume that a better paradigm is needed to reveal any potential visuo-tactile convergence. However, the present study identified brain regions that may subserve the discrimination of different intensities of visual roughness. This finding may contribute to elucidate the neural mechanisms related to the visual roughness perception in the human brain.     

Somatic and motor components of action simulation

Seminal studies in monkeys report that the viewing of actions performed by other individuals activates frontal and parietal cortical areas typically involved in action planning and execution. That mirroring actions might rely on both motor and somatosensory components is suggested by reports that action observation and execution increase neural activity in motor and in somatosensory areas. This occurs not only during observation of naturalistic movements but also during the viewing of biomechanically impossible movements that tap the afferent component of action, possibly by eliciting strong somatic feelings in the onlooker. Although somatosensory feedback is inherently linked to action execution, information on the possible causative role of frontal and parietal cortices in simulating motor and sensory action components is lacking. By combining low-frequency repetitive and single-pulse transcranial magnetic stimulation, we found that virtual lesions of ventral premotor cortex (vPMc) and primary somatosensory cortex (S1) suppressed mirror motor facilitation contingent upon observation of possible and impossible movements, respectively. In contrast, virtual lesions of primary motor cortex did not influence mirror motor facilitation. The reported double dissociation suggests that vPMc and S1 play an active, differential role in simulating efferent and afferent components of observed actions.     

Social Attention,Affective Arousal and Empathy in Men with Klinefelter Syndrome (47,XXY): Evidence from Eyetracking and Skin Conductance

Individuals with an extra X chromosome (Klinefelter syndrome) are at risk for problems in social functioning and have an increased vulnerability for autism traits. In the search for underlying mechanisms driving this increased risk, this study focused on social attention, affective arousal and empathy. Seventeen adults with XXY and 20 non-clinical controls participated in this study. Eyetracking was used to investigate social attention, as expressed in visual scanning patterns in response to the viewing of empathy evoking video clips. Skin conductance levels, reflecting affective arousal, were recorded continuously during the clips as well. Empathic skills, i.e. participants'' understanding of own and others'' emotions in response to the clips was also assessed. Results showed reduced empathic understanding, decreased visual fixation to the eye region, but increased affective arousal in individuals with Klinefelter syndrome. We conclude that individuals with XXY tend to avoid the eye region. Considering the increased affective arousal, we speculate that this attentional deployment strategy may not be sufficient to successfully downregulate affective hyper-responsivity. As increased affective arousal was related to reduced empathic ability, we hypothesize that own affective responses to social cues play an important role in difficulties in understanding the feelings and intentions of others. This knowledge may help in the identification of risk factors for psychopathology and targets for treatment.     

Facilitation of Visual Perception in Head Direction: Visual Attention Modulation Based on Head Direction

People usually see things using frontal viewing, and avoid lateral viewing (or eccentric gaze) where the directions of the head and eyes are largely different. Lateral viewing interferes with attentive visual search performance, probably because the head is directed away from the target and/or because the head and eyes are misaligned. In this study, we examined which of these factors is the primary one for interference by conducting a visual identification experiment where a target was presented in the peripheral visual field. The critical manipulation was the participants’ head direction and fixation position: the head was directed to the fixation location, the target position, or the opposite side of the fixation. The performance was highest when the head was directed to the target position even when there was misalignment of the head and eye, suggesting that visual perception can be influenced by both head direction and fixation position.     

Personality-related differences in recognition of facial emotional expression and cortical evoked potentials

A correlation between some characteristics of the visual evoked potentials and individual personality traits (by the Kettell scale) was revealed in 40 healthy subjects when they recognized facial expressions of anger and fear. As compared to emotionally stable subjects, emotionally unstable subjects had shorter latencies of evoked potentials and suppressed late negativity in the occipital and temporal areas. In contrast, amplitude of these waves in the frontal areas was increased. In emotionally stable group of subjects differences in the evoked potentials related to emotional expressions were evident throughout the whole signal processing beginning from the early sensory stage (P1 wave). In emotionally unstable group differences in the evoked potentials related to recognized emotional expressions developed later. Sensitivity of the evoked potentials to emotional salience of faces was also more pronounced in the emotionally stable group. The involvement of the frontal cortex, amygdala, and the anterior cingulate cortex in the development of individual features of recognition of facial expressions of anger and fear is discussed.     

Changes in salivary testosterone concentrations and subsequent voluntary squat performance following the presentation of short video clips

Previous studies have shown that visual images can produce rapid changes in testosterone concentrations. We explored the acute effects of video clips on salivary testosterone and cortisol concentrations and subsequent voluntary squat performance in highly trained male athletes (n=12). Saliva samples were collected on 6 occasions immediately before and 15 min after watching a brief video clip (approximately 4 min in duration) on a computer screen. The watching of a sad, erotic, aggressive, training motivational, humorous or a neutral control clip was randomised. Subjects then performed a squat workout aimed at producing a 3 repetition maximum (3RM) lift. Significant (P<0.001) relative (%) increases in testosterone concentrations were noted with watching the erotic, humorous, aggressive and training videos (versus control and sad), with testosterone decreasing significantly (versus control) after the sad clip. The aggressive video also produced an elevated cortisol response (% change) and more so than the control and humorous videos (P<0.001). A significant (P<0.003) improvement in 3RM performance was noted after the erotic, aggressive and training clips (versus control). A strong within-individual correlation (mean r=0.85) was also noted between the relative changes in testosterone and the 3RM squats across all video sessions (P<0.001). In conclusion, different video clips were associated with different changes in salivary free hormone concentrations and the relative changes in testosterone closely mapped 3RM squat performance in a group of highly trained males. Thus, speculatively, using short video presentations in the pre-workout environment offers an opportunity for understanding the outcomes of hormonal change, athlete behaviour and subsequent voluntary performance.     

The neural fate of consciously perceived and missed events in the attentional blink

Cognitive models of attention propose that visual perception is a product of two stages of visual processing: early operations permit rapid initial categorization of the visual world, while later attention-demanding capacity-limited stages are necessary for the conscious report of the stimuli. Here we used the attentional blink paradigm and fMRI to neurally distinguish these two stages of vision. Subjects detected a face target and a scene target presented rapidly among distractors at fixation. Although the second, scene target frequently went undetected by the subjects, it nonetheless activated regions of the medial temporal cortex involved in high-level scene representations, the parahippocampal place area (PPA). This PPA activation was amplified when the stimulus was consciously perceived. By contrast, the frontal cortex was activated only when scenes were successfully reported. These results suggest that medial temporal cortex permits rapid categorization of the visual input, while the frontal cortex is part of a capacity-limited attentional bottleneck to conscious report.     

Monitoring of selections of visual stimuli and the primate frontal cortex.

This investigation shows that lesions confined to the middle sector of the dorsolateral frontal cortex, i.e. cytoarchitectonic areas 46 and 9, cause a striking impairment in the ability of non-human primates to recall which one from a set of stimuli they chose, without in any way affecting their ability to recognize that they had previously seen those stimuli. By contrast, lesions placed within the adjacent posterior dorsolateral frontal cortex affect neither recognition of visual stimuli nor recall of prior choices. These findings delineate the mid-dorsolateral frontal cortex as a critical component of a neural system mediating the monitoring of self-generated responses.     

"Slight" of hand: the processing of visually degraded gestures with speech

Co-speech hand gestures influence language comprehension. The present experiment explored what part of the visual processing system is optimized for processing these gestures. Participants viewed short video clips of speech and gestures (e.g., a person saying "chop" or "twist" while making a chopping gesture) and had to determine whether the two modalities were congruent or incongruent. Gesture videos were designed to stimulate the parvocellular or magnocellular visual pathways by filtering out low or high spatial frequencies (HSF versus LSF) at two levels of degradation severity (moderate and severe). Participants were less accurate and slower at processing gesture and speech at severe versus moderate levels of degradation. In addition, they were slower for LSF versus HSF stimuli, and this difference was most pronounced in the severely degraded condition. However, exploratory item analyses showed that the HSF advantage was modulated by the range of motion and amount of motion energy in each video. The results suggest that hand gestures exploit a wide range of spatial frequencies, and depending on what frequencies carry the most motion energy, parvocellular or magnocellular visual pathways are maximized to quickly and optimally extract meaning.     

Dissociation between ventral and dorsal fMRI activation during object and action recognition

Neuropsychological case studies suggest the existence of two functionally separate visual streams: the ventral pathway, central for object recognition; and the dorsal pathway, engaged in visually guided actions. However, a clear dissociation between the functions of the two streams has not been decisively shown in intact humans. In this study, we demonstrate dissociation between dorsal and ventral fMRI activation patterns during observation of object manipulation video clips. Parietal areas, such as anterior intraparietal sulcus (aIPS) display grasp viewing-dependent adaptation (i.e., fMR adaptation during repeated viewing of the same object-grasping movement) as well as a contralateral preference for the viewed manipulating hand. Ventral regions, such as the fusiform gyrus, show similar characteristics (i.e., adaptation, contralateral preference), but these depend on object identity. Our results support the hypothesized functional specialization in the visual system and suggest that parietal areas (such as aIPS) are engaged in action recognition, as well as in action planning.     

Visual Function and Cortical Organization in Carriers of Blue Cone Monochromacy

Carriers of blue cone monochromacy have fewer cone photoreceptors than normal. Here we examine how this disruption at the level of the retina affects visual function and cortical organization in these individuals. Visual resolution and contrast sensitivity was measured at the preferred retinal locus of fixation and visual resolution was tested at two eccentric locations (2.5° and 8°) with spectacle correction only. Adaptive optics corrected resolution acuity and cone spacing were simultaneously measured at several locations within the central fovea with adaptive optics scanning laser ophthalmoscopy (AOSLO). Fixation stability was assessed by extracting eye motion data from AOSLO videos. Retinotopic mapping using fMRI was carried out to estimate the area of early cortical regions, including that of the foveal confluence. Without adaptive optics correction, BCM carriers appeared to have normal visual function, with normal contrast sensitivity and visual resolution, but with AO-correction, visual resolution was significantly worse than normal. This resolution deficit is not explained by cone loss alone and is suggestive of an associated loss of retinal ganglion cells. However, despite evidence suggesting a reduction in the number of retinal ganglion cells, retinotopic mapping showed no reduction in the cortical area of the foveal confluence. These results suggest that ganglion cell density may not govern the foveal overrepresentation in the cortex. We propose that it is not the number of afferents, but rather the content of the information relayed to the cortex from the retina across the visual field that governs cortical magnification, as under normal viewing conditions this information is similar in both BCM carriers and normal controls.     

Attention to stimulus features shifts spectral tuning of V4 neurons during natural vision

Previous neurophysiological studies suggest that attention can alter the baseline or gain of neurons in extrastriate visual areas but that it cannot change tuning. This suggests that neurons in visual cortex function as labeled lines whose meaning does not depend on task demands. To test this common assumption, we used a system identification approach to measure spatial frequency and orientation tuning in area V4 during two attentionally demanding visual search tasks, one that required fixation and one that allowed free viewing during search. We found that spatial attention modulates response baseline and gain but does not alter tuning, consistent with previous reports. In contrast, feature-based attention often shifts neuronal tuning. These tuning shifts are inconsistent with the labeled-line model and tend to enhance responses to stimulus features that distinguish the search target. Our data suggest that V4 neurons behave as matched filters that are dynamically tuned to optimize visual search.     

Cortical positive potentials in the period of eye fixation prior to saccades and antisaccades

The EEG of 10 right-handed healthy subjects preceding saccade and antisaccade with mean values of latency in the eye fixations period were selected and averaged. The positive potential P2 appearing on the fixation stimuly switching on and slow positive wave following after it were more prominent before antisaccades than normal saccades. Space-temporal analyses of presaccadic potentials showed that right frontal cortex was activated more before antisaccades. These findings suggest that right cortical hemisphere dominate in spatial attention and inhibition of automatic saccades to visual stimuli in the period of antisaccades preparing. During the period of central fixations "intermediate" positivity potentials, developing in 600-400 ms prior to saccade or antisaccade onset, were find out. These potentials were predominantly recorded in the left frontal and frontosagittal cortical areas. The obtained evidence suggest that "intermediate" positive potentials a period related to the process of motor attention, anticipation and decision making in the period of eyes fixation.     

Predicting the stream of consciousness from activity in human visual cortex

Can the rapid stream of conscious experience be predicted from brain activity alone? Recently, spatial patterns of activity in visual cortex have been successfully used to predict feature-specific stimulus representations for both visible and invisible stimuli. However, because these studies examined only the prediction of static and unchanging perceptual states during extended periods of stimulation, it remains unclear whether activity in early visual cortex can also predict the rapidly and spontaneously changing stream of consciousness. Here, we used binocular rivalry to induce frequent spontaneous and stochastic changes in conscious experience without any corresponding changes in sensory stimulation, while measuring brain activity with fMRI. Using information that was present in the multivariate pattern of responses to stimulus features, we could accurately predict, and therefore track, participants' conscious experience from the fMRI signal alone while it underwent many spontaneous changes. Prediction in primary visual cortex primarily reflected eye-based signals, whereas prediction in higher areas reflected the color of the percept. Furthermore, accurate prediction during binocular rivalry could be established with signals recorded during stable monocular viewing, showing that prediction generalized across viewing conditions and did not require or rely on motor responses. It is therefore possible to predict the dynamically changing time course of subjective experience with only brain activity.     

Temporal evolution of a decision-making process in medial premotor cortex

The events linking sensory discrimination to motor action remain unclear. It is not known, for example, whether the motor areas of the frontal lobe receive the result of the discrimination process from other areas or whether they actively participate in it. To investigate this, we trained monkeys to discriminate between two mechanical vibrations applied sequentially to the fingertips; here subjects had to recall the first vibration, compare it to the second one, and indicate with a hand/arm movement which of the two vibrations had the higher frequency. We recorded the activity of single neurons in medial premotor cortex (MPC) and found that their responses correlate with the diverse stages of the discrimination process. Thus, activity in MPC reflects the temporal evolution of the decision-making process leading to action selection during this perceptual task.