Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding

While much has been learned regarding the neural substrates supporting episodic encoding using highly controlled experimental protocols, relatively little is known regarding the neural bases of episodic encoding of real-world events. In an effort to examine this issue, we measured fMRI activity while observers viewed a novel TV sitcom. Three weeks later, subsequent memory (SM) for the narrative content of movie events was assessed. We analyzed the encoding data for intersubject correlations (ISC) based on subjects' subsequent memory (ISC-SM) performance to identify brain regions whose BOLD response is significantly more correlated across subjects during portions of the movie that are successfully as compared to unsuccessfully encoded. These regions include the parahippocampal gyrus, superior temporal gyrus, anterior temporal poles, and the temporal-parietal junction. Further analyses reveal (1) that these correlated regions can display distinct activation profiles and (2) that the results seen with the ISC-SM analysis are complementary to more traditional linear models and allow analysis of complex time course data. Thus, the ISC-SM analysis extends traditional subsequent memory findings to a rich, dynamic and more ecologically valid situation.     

Synchronization of spontaneous eyeblinks while viewing video stories

Blinks are generally suppressed during a task that requires visual attention and tend to occur immediately before or after the task when the timing of its onset and offset are explicitly given. During the viewing of video stories, blinks are expected to occur at explicit breaks such as scene changes. However, given that the scene length is unpredictable, there should also be appropriate timing for blinking within a scene to prevent temporal loss of critical visual information. Here, we show that spontaneous blinks were highly synchronized between and within subjects when they viewed the same short video stories, but were not explicitly tied to the scene breaks. Synchronized blinks occurred during scenes that required less attention such as at the conclusion of an action, during the absence of the main character, during a long shot and during repeated presentations of a similar scene. In contrast, blink synchronization was not observed when subjects viewed a background video or when they listened to a story read aloud. The results suggest that humans share a mechanism for controlling the timing of blinks that searches for an implicit timing that is appropriate to minimize the chance of losing critical information while viewing a stream of visual events.     

Visual fatigue evaluation of field sequential stereoscopic 3D display using near point distance

The purpose of this paper was to evaluate quantitatively visual fatigue encountered in viewing stereoscopic 3D television. By examining the difference in the visual function involved in watching 2D and stereoscopic 3D displays, we could extract only the effect on visual perception in stereoscopic 3D television. Since the visual fatigue with the 3D display may be caused by the discrepancy between convergence and accommodation, the near point distances were measured by an Accommodo Polyrecorder, where Landolt's ring was pursued with binocularity. Five subjects, all employees of NEC Corporation, participated in the experiment. All subjects had normal or corrected-to-normal vision. Two of the subjects watched a 3D video movie. After a week, they watched another video movie in 2D. The rest of them watched the movies in reverse order. One video movie viewing requires about 120 minutes. The near point distances were measured before and after watching for 30, 60, 90 and 120 minutes. The 3D television system used here is a 3D video disc system with field sequence using glasses with liquid crystal shutters. In the case of 2D, each subject wore the glasses whose shutters were kept open. In the experiment, instruction to watch the video movies in a relaxed posture was given. There was a significant effect in regard to 3D/2D treatments on the change rates of the near point distance measured after watching every 30 minutes to one before watching. In addition, the change rates in 3D watching after 60, 90 and 120 minutes were significantly increased, compared with the corresponding rates in 2D.(ABSTRACT TRUNCATED AT 250 WORDS)     

Memory and prediction in natural gaze control

In addition to stimulus properties and task factors, memory is an important determinant of the allocation of attention and gaze in the natural world. One way that the role of memory is revealed is by predictive eye movements. Both smooth pursuit and saccadic eye movements demonstrate predictive effects based on previous experience. We have previously shown that unskilled subjects make highly accurate predictive saccades to the anticipated location of a ball prior to a bounce in a virtual racquetball setting. In this experiment, we examined this predictive behaviour. We asked whether the period after the bounce provides subjects with visual information about the ball trajectory that is used to programme the pursuit movement initiated when the ball passes through the fixation point. We occluded a 100 ms period of the ball''s trajectory immediately after the bounce, and found very little effect on the subsequent pursuit movement. Subjects did not appear to modify their strategy to prolong the fixation. Neither were we able to find an effect on interception performance. Thus, it is possible that the occluded trajectory information is not critical for subsequent pursuit, and subjects may use an estimate of the ball''s trajectory to programme pursuit. These results provide further support for the role of memory in eye movements.     

Field of attention for instantaneous object recognition

Background

Instantaneous object discrimination and categorization are fundamental cognitive capacities performed with the guidance of visual attention. Visual attention enables selection of a salient object within a limited area of the visual field; we referred to as “field of attention” (FA). Though there is some evidence concerning the spatial extent of object recognition, the following questions still remain unknown: (a) how large is the FA for rapid object categorization, (b) how accuracy of attention is distributed over the FA, and (c) how fast complex objects can be categorized when presented against backgrounds formed by natural scenes.

Methodology/Principal Findings

To answer these questions, we used a visual perceptual task in which subjects were asked to focus their attention on a point while being required to categorize briefly flashed (20 ms) photographs of natural scenes by indicating whether or not these contained an animal. By measuring the accuracy of categorization at different eccentricities from the fixation point, we were able to determine the spatial extent and the distribution of accuracy over the FA, as well as the speed of categorizing objects using stimulus onset asynchrony (SOA). Our results revealed that subjects are able to rapidly categorize complex natural images within about 0.1 s without eye movement, and showed that the FA for instantaneous image categorization covers a visual field extending 20°×24°, and accuracy was highest (>90%) at the center of FA and declined with increasing eccentricity.

Conclusions/Significance

In conclusion, human beings are able to categorize complex natural images at a glance over a large extent of the visual field without eye movement.     

Overt attention and context factors: the impact of repeated presentations, image type, and individual motivation

The present study investigated the dynamic of the attention focus during observation of different categories of complex scenes and simultaneous consideration of individuals' memory and motivational state. We repeatedly presented four types of complex visual scenes in a pseudo-randomized order and recorded eye movements. Subjects were divided into groups according to their motivational disposition in terms of action orientation and individual rating of scene interest.Statistical analysis of eye-tracking data revealed that the attention focus successively became locally expressed by increasing fixation duration; decreasing saccade length, saccade frequency, and single subject's fixation distribution over images; and increasing inter-subject variance of fixation distributions. The validity of these results was supported by verbal reports. This general tendency was weaker for the group of subjects who rated the image set as interesting as compared to the other group. Additionally, effects were partly mediated by subjects' motivational disposition. Finally, we found a generally strong impact of image type on eye movement parameters. We conclude that motivational tendencies linked to personality as well as individual preferences significantly affected viewing behaviour. Hence, it is important and fruitful to consider inter-individual differences on the level of motivation and personality traits within investigations of attention processes. We demonstrate that future studies on memory's impact on overt attention have to deal appropriately with several aspects that had been out of the research focus until now.     

Task attention facilitates learning of task-irrelevant stimuli

Attention plays a fundamental role in visual learning and memory. One highly established principle of visual attention is that the harder a central task is, the more attentional resources are used to perform the task and the smaller amount of attention is allocated to peripheral processing because of limited attention capacity. Here we show that this principle holds true in a dual-task setting but not in a paradigm of task-irrelevant perceptual learning. In Experiment 1, eight participants were asked to identify either bright or dim number targets at the screen center and to remember concurrently presented scene backgrounds. Their recognition performances for scenes paired with dim/hard targets were worse than those for scenes paired with bright/easy targets. In Experiment 2, eight participants were asked to identify either bright or dim letter targets at the screen center while a task-irrelevant coherent motion was concurrently presented in the background. After five days of training on letter identification, participants improved their motion sensitivity to the direction paired with hard/dim targets improved but not to the direction paired with easy/bright targets. Taken together, these results suggest that task-irrelevant stimuli are not subject to the attentional control mechanisms that task-relevant stimuli abide.     

Implicit learning and implicit memory for odors: the influence of odor identification and retention time

One hundred and fifty-two subjects, divided into eight groups, were exposed to a room with a low concentration of either orange or lavender and to an odorless room. In a careful double-blind procedure, neither the subjects nor the experimenters were made aware of the presence of the odors in the experimental conditions. Later they were asked to indicate how well each of 12 odor stimuli, including the experimental and control odors, befitted each of 12 visual contexts, including the exposure rooms. At the end of this session they rated the pleasantness and the familiarity of the odors, and identified them by name. Finally they were debriefed and asked specifically whether they had perceived the experimental odors anywhere in the building. The results of four subjects who answered positively to the latter question were omitted. The results confirm the earlier finding that non-identifiers implicitly link odor and exposure room, whereas identifiers do not show such a link. It is suggested that episodic information is an essential constituent of olfactory memory and that its function is comparable to that of form and structure in visual and auditory memory systems.     

Eye Tracking,Cortisol, and a Sleep vs. Wake Consolidation Delay: Combining Methods to Uncover an Interactive Effect of Sleep and Cortisol on Memory

Although rises in cortisol can benefit memory consolidation, as can sleep soon after encoding, there is currently a paucity of literature as to how these two factors may interact to influence consolidation. Here we present a protocol to examine the interactive influence of cortisol and sleep on memory consolidation, by combining three methods: eye tracking, salivary cortisol analysis, and behavioral memory testing across sleep and wake delays. To assess resting cortisol levels, participants gave a saliva sample before viewing negative and neutral objects within scenes. To measure overt attention, participants’ eye gaze was tracked during encoding. To manipulate whether sleep occurred during the consolidation window, participants either encoded scenes in the evening, slept overnight, and took a recognition test the next morning, or encoded scenes in the morning and remained awake during a comparably long retention interval. Additional control groups were tested after a 20 min delay in the morning or evening, to control for time-of-day effects. Together, results showed that there is a direct relation between resting cortisol at encoding and subsequent memory, only following a period of sleep. Through eye tracking, it was further determined that for negative stimuli, this beneficial effect of cortisol on subsequent memory may be due to cortisol strengthening the relation between where participants look during encoding and what they are later able to remember. Overall, results obtained by a combination of these methods uncovered an interactive effect of sleep and cortisol on memory consolidation.     

Eye Exercises Enhance Accuracy and Letter Recognition,but Not Reaction Time,in a Modified Rapid Serial Visual Presentation Task

Eye exercises have been prescribed to resolve a multitude of eye-related problems. However, studies on the efficacy of eye exercises are lacking, mainly due to the absence of simple assessment tools in the clinic. Because similar regions of the brain are responsible for eye movements and visual attention, we used a modified rapid serial visual presentation (RSVP) to assess any measurable effect of short-term eye exercise in improvements within these domains. In the present study, twenty subjects were equally divided into control and experimental groups, each of which performed a pre-training RSVP assessment where target letters, to which subjects were asked to respond to by pressing a spacebar, were serially and rapidly presented. Response time to target letters, accuracy of correctly responding to target letters, and correct identification of target letters in each of 12 sessions was measured. The experimental group then performed active eye exercises, while the control group performed a task that minimized eye movements for 18.5 minutes. A final post-training RSVP assessment was performed by both groups and response time, accuracy, and letter identification were compared between and within subject groups both pre- and post-training. Subjects who performed eye exercises were more accurate in responding to target letters separated by one distractor and in letter identification in the post-training RSVP assessment, while latency of responses were unchanged between and within groups. This suggests that eye exercises may prove useful in enhancing cognitive performance on tasks related to attention and memory over a very brief course of training, and RSVP may be a useful measure of this efficacy. Further research is needed on eye exercises to determine whether they are an effective treatment for patients with cognitive and eye-related disorders.     

Mesmerizing memories: brain substrates of episodic memory suppression in posthypnotic amnesia

Two groups of participants, one susceptible to posthypnotic amnesia (PHA) and the other not, viewed a movie. A week later, they underwent hypnosis in the fMRI scanner and received a suggestion to forget the movie details after hypnosis until receiving a reversal cue. The participants were tested twice for memory for the movie and for the context in which it was shown, under the posthypnotic suggestion and after its reversal, while their brain was scanned. The PHA group showed reduced memory for movie but not for context while under suggestion. Activity in occipital, temporal, and prefrontal areas differed among the groups, and, in the PHA group, between suggestion and reversal conditions. We propose that whereas some of these regions subserve retrieval of long-term episodic memory, others are involved in inhibiting retrieval, possibly already in a preretrieval monitoring stage. Similar mechanisms may also underlie other forms of functional amnesia.     

Is Attention Based on Spatial Contextual Memory Preferentially Guided by Low Spatial Frequency Signals?

A popular model of visual perception states that coarse information (carried by low spatial frequencies) along the dorsal stream is rapidly transmitted to prefrontal and medial temporal areas, activating contextual information from memory, which can in turn constrain detailed input carried by high spatial frequencies arriving at a slower rate along the ventral visual stream, thus facilitating the processing of ambiguous visual stimuli. We were interested in testing whether this model contributes to memory-guided orienting of attention. In particular, we asked whether global, low-spatial frequency (LSF) inputs play a dominant role in triggering contextual memories in order to facilitate the processing of the upcoming target stimulus. We explored this question over four experiments. The first experiment replicated the LSF advantage reported in perceptual discrimination tasks by showing that participants were faster and more accurate at matching a low spatial frequency version of a scene, compared to a high spatial frequency version, to its original counterpart in a forced-choice task. The subsequent three experiments tested the relative contributions of low versus high spatial frequencies during memory-guided covert spatial attention orienting tasks. Replicating the effects of memory-guided attention, pre-exposure to scenes associated with specific spatial memories for target locations (memory cues) led to higher perceptual discrimination and faster response times to identify targets embedded in the scenes. However, either high or low spatial frequency cues were equally effective; LSF signals did not selectively or preferentially contribute to the memory-driven attention benefits to performance. Our results challenge a generalized model that LSFs activate contextual memories, which in turn bias attention and facilitate perception.     

Eye movements predict recollective experience

Previously encountered stimuli can bring to mind a vivid memory of the episodic context in which the stimulus was first experienced ("remembered" stimuli), or can simply seem familiar ("known" stimuli). Past studies suggest that more attentional resources are required to encode stimuli that are subsequently remembered than known. However, it is unclear if the attentional resources are distributed differently during encoding and recognition of remembered and known stimuli. Here, we record eye movements while participants encode photos, and later while indicating whether the photos are remembered, known or new. Eye fixations were more clustered during both encoding and recognition of remembered photos relative to known photos. Thus, recognition of photos that bring to mind a vivid memory for the episodic context in which they were experienced is associated with less distributed overt attention during encoding and recognition. The results suggest that remembering is related to encoding of a few distinct details of a photo rather than the photo as a whole. In turn, during recognition remembering may be trigged by enhanced memory for the salient details of the photos.     

Categorization of Natural Dynamic Audiovisual Scenes

This work analyzed the perceptual attributes of natural dynamic audiovisual scenes. We presented thirty participants with 19 natural scenes in a similarity categorization task, followed by a semi-structured interview. The scenes were reproduced with an immersive audiovisual display. Natural scene perception has been studied mainly with unimodal settings, which have identified motion as one of the most salient attributes related to visual scenes, and sound intensity along with pitch trajectories related to auditory scenes. However, controlled laboratory experiments with natural multimodal stimuli are still scarce. Our results show that humans pay attention to similar perceptual attributes in natural scenes, and a two-dimensional perceptual map of the stimulus scenes and perceptual attributes was obtained in this work. The exploratory results show the amount of movement, perceived noisiness, and eventfulness of the scene to be the most important perceptual attributes in naturalistically reproduced real-world urban environments. We found the scene gist properties openness and expansion to remain as important factors in scenes with no salient auditory or visual events. We propose that the study of scene perception should move forward to understand better the processes behind multimodal scene processing in real-world environments. We publish our stimulus scenes as spherical video recordings and sound field recordings in a publicly available database.     

Orienting attention based on long-term memory experience

Attentional orienting and memory are intrinsically bound, but their interaction has rarely been investigated. Here we introduce an experimental paradigm using naturalistic scenes to investigate how long-term memory can guide spatial attention and thereby enhance identification of events in the perceptual domain. In the task, stable memories of objects embedded within complex scenes guide spatial orienting. We compared the behavioral effects and neural systems of memory-guided orienting with those in a more traditional attention-orienting task in which transient spatial cues guide attention. Memory-guided attention operated within surprisingly short intervals and conferred reliable and sizeable advantages for detection of objects embedded in scenes. Event-related functional magnetic resonance imaging showed that memory-guided attention involves the interaction between brain areas participating in retrieval of memories for spatial context with the parietal-frontal network for visual spatial orienting. Activity in the hippocampus was specifically engaged in memory-guided spatial attention and correlated with the ensuing behavioral advantage.     

Odors: implicit memory and performance effects.

In order to assess the influence of odors on human performance and implicit memory for odors, 108 subjects completed a variety of tests in weakly scented (jasmine, lavender or odorless) rooms without having been made aware of the odor. After a 30 min interval the subjects were shown slides of different surroundings, including the room they had been in, and were requested to rate how well a set of 12 odors, including a blank, would fit to these surroundings. Half of these contexts contained visual cues related to two of the presented odors (leather and coffee). After the rating of fit the subjects had to rate the odors for pleasantness, were asked to identify the odors with their correct names and to tell where and when they had last smelled these odors. One subject remembered smelling the odor (jasmine) in the room and was discarded from the analysis of the results for the rating of fit. None of the others reported recollection of the experimental odors. The results showed that in general jasmine had a negative and lavender a positive effect on test performance. If an odor-related visual cue was present in the context, the related odor was always rated highest in fit to that context. Furthermore, the subjects working in rooms with an odor subsequently assigned this odor to the visual context of that room to a significantly higher degree than subjects working in rooms with different odors. Since none of the subjects reported that they had smelled the odor in the rooms where performance testing took place, it was concluded that the memory for these odors was implicit. Further analysis showed that such memory was only found in subjects who were unable to supply the right name for the odor. The possible consequences of this latter finding for understanding the relationship between sensory (episodic) and semantic odor memory are discussed.     

Predicting the eye fixation locations in the gray scale images in the visual scenes with different semantic contents

In recent years, there has been considerable interest in visual attention models (saliency map of visual attention). These models can be used to predict eye fixation locations, and thus will have many applications in various fields which leads to obtain better performance in machine vision systems. Most of these models need to be improved because they are based on bottom-up computation that does not consider top-down image semantic contents and often does not match actual eye fixation locations. In this study, we recorded the eye movements (i.e., fixations) of fourteen individuals who viewed images which consist natural (e.g., landscape, animal) and man-made (e.g., building, vehicles) scenes. We extracted the fixation locations of eye movements in two image categories. After extraction of the fixation areas (a patch around each fixation location), characteristics of these areas were evaluated as compared to non-fixation areas. The extracted features in each patch included the orientation and spatial frequency. After feature extraction phase, different statistical classifiers were trained for prediction of eye fixation locations by these features. This study connects eye-tracking results to automatic prediction of saliency regions of the images. The results showed that it is possible to predict the eye fixation locations by using of the image patches around subjects’ fixation points.     

Beyond the Movie Screen: An Antarctic Adventure

Movies depicting science-related issues often capture the attention of today's youth. As an instructional tool, movies can take us beyond the drama and action and thrilling scenes. In this article we share our experiences of using the movie Eight Below as a centerpiece for developing high school students’ understanding of basic chemistry concepts. In addition to the science concepts developed using the context of the movie, we engaged students in thinking about the ethical dilemmas embedded in the movie.     

Emergence of visual saliency from natural scenes via context-mediated probability distributions coding

Visual saliency is the perceptual quality that makes some items in visual scenes stand out from their immediate contexts. Visual saliency plays important roles in natural vision in that saliency can direct eye movements, deploy attention, and facilitate tasks like object detection and scene understanding. A central unsolved issue is: What features should be encoded in the early visual cortex for detecting salient features in natural scenes? To explore this important issue, we propose a hypothesis that visual saliency is based on efficient encoding of the probability distributions (PDs) of visual variables in specific contexts in natural scenes, referred to as context-mediated PDs in natural scenes. In this concept, computational units in the model of the early visual system do not act as feature detectors but rather as estimators of the context-mediated PDs of a full range of visual variables in natural scenes, which directly give rise to a measure of visual saliency of any input stimulus. To test this hypothesis, we developed a model of the context-mediated PDs in natural scenes using a modified algorithm for independent component analysis (ICA) and derived a measure of visual saliency based on these PDs estimated from a set of natural scenes. We demonstrated that visual saliency based on the context-mediated PDs in natural scenes effectively predicts human gaze in free-viewing of both static and dynamic natural scenes. This study suggests that the computation based on the context-mediated PDs of visual variables in natural scenes may underlie the neural mechanism in the early visual cortex for detecting salient features in natural scenes.     

Iterative Fragmentation of Cognitive Maps in a Visual Imagery Task

It remains unclear whether spontaneous eye movements during visual imagery reflect the mental generation of a visual image (i.e. the arrangement of the component parts of a mental representation). To address this specificity, we recorded eye movements in an imagery task and in a phonological fluency (non-imagery) task, both consisting in naming French towns from long-term memory. Only in the condition of visual imagery the spontaneous eye positions reflected the geographic position of the towns evoked by the subjects. This demonstrates that eye positions closely reflect the mapping of mental images. Advanced analysis of gaze positions using the bi-dimensional regression model confirmed the spatial correlation of gaze and towns’ locations in every single individual in the visual imagery task and in none of the individuals when no imagery accompanied memory retrieval. In addition, the evolution of the bi-dimensional regression’s coefficient of determination revealed, in each individual, a process of generating several iterative series of a limited number of towns mapped with the same spatial distortion, despite different individual order of towns’ evocation and different individual mappings. Such consistency across subjects revealed by gaze (the mind’s eye) gives empirical support to theories postulating that visual imagery, like visual sampling, is an iterative fragmented processing.