Psychophysical tests of the hypothesis of a bottom-up saliency map in primary visual cortex

A unique vertical bar among horizontal bars is salient and pops out perceptually. Physiological data have suggested that mechanisms in the primary visual cortex (V1) contribute to the high saliency of such a unique basic feature, but indicated little regarding whether V1 plays an essential or peripheral role in input-driven or bottom-up saliency. Meanwhile, a biologically based V1 model has suggested that V1 mechanisms can also explain bottom-up saliencies beyond the pop-out of basic features, such as the low saliency of a unique conjunction feature such as a red vertical bar among red horizontal and green vertical bars, under the hypothesis that the bottom-up saliency at any location is signaled by the activity of the most active cell responding to it regardless of the cell's preferred features such as color and orientation. The model can account for phenomena such as the difficulties in conjunction feature search, asymmetries in visual search, and how background irregularities affect ease of search. In this paper, we report nontrivial predictions from the V1 saliency hypothesis, and their psychophysical tests and confirmations. The prediction that most clearly distinguishes the V1 saliency hypothesis from other models is that task-irrelevant features could interfere in visual search or segmentation tasks which rely significantly on bottom-up saliency. For instance, irrelevant colors can interfere in an orientation-based task, and the presence of horizontal and vertical bars can impair performance in a task based on oblique bars. Furthermore, properties of the intracortical interactions and neural selectivities in V1 predict specific emergent phenomena associated with visual grouping. Our findings support the idea that a bottom-up saliency map can be at a lower visual area than traditionally expected, with implications for top-down selection mechanisms.     

Visual saliency and spike timing in the ventral visual pathway.

Visual saliency is a fundamental yet hard to define property of objects or locations in the visual world. In a context where objects and their representations compete to dominate our perception, saliency can be thought of as the "juice" that makes objects win the race. It is often assumed that saliency is extracted and represented in an explicit saliency map, which serves to determine the location of spatial attention at any given time. It is then by drawing attention to a salient object that it can be recognized or categorized. I argue against this classical view that visual "bottom-up" saliency automatically recruits the attentional system prior to object recognition. A number of visual processing tasks are clearly performed too fast for such a costly strategy to be employed. Rather, visual attention could simply act by biasing a saliency-based object recognition system. Under natural conditions of stimulation, saliency can be represented implicitly throughout the ventral visual pathway, independent of any explicit saliency map. At any given level, the most activated cells of the neural population simply represent the most salient locations. The notion of saliency itself grows increasingly complex throughout the system, mostly based on luminance contrast until information reaches visual cortex, gradually incorporating information about features such as orientation or color in primary visual cortex and early extrastriate areas, and finally the identity and behavioral relevance of objects in temporal cortex and beyond. Under these conditions the object that dominates perception, i.e. the object yielding the strongest (or the first) selective neural response, is by definition the one whose features are most "salient"--without the need for any external saliency map. In addition, I suggest that such an implicit representation of saliency can be best encoded in the relative times of the first spikes fired in a given neuronal population. In accordance with our subjective experience that saliency and attention do not modify the appearance of objects, the feed-forward propagation of this first spike wave could serve to trigger saliency-based object recognition outside the realm of awareness, while conscious perceptions could be mediated by the remaining discharges of longer neuronal spike trains.     

Primary Visual Cortex as a Saliency Map: A Parameter-Free Prediction and Its Test by Behavioral Data

It has been hypothesized that neural activities in the primary visual cortex (V1) represent a saliency map of the visual field to exogenously guide attention. This hypothesis has so far provided only qualitative predictions and their confirmations. We report this hypothesis’ first quantitative prediction, derived without free parameters, and its confirmation by human behavioral data. The hypothesis provides a direct link between V1 neural responses to a visual location and the saliency of that location to guide attention exogenously. In a visual input containing many bars, one of them saliently different from all the other bars which are identical to each other, saliency at the singleton’s location can be measured by the shortness of the reaction time in a visual search for singletons. The hypothesis predicts quantitatively the whole distribution of the reaction times to find a singleton unique in color, orientation, and motion direction from the reaction times to find other types of singletons. The prediction matches human reaction time data. A requirement for this successful prediction is a data-motivated assumption that V1 lacks neurons tuned simultaneously to color, orientation, and motion direction of visual inputs. Since evidence suggests that extrastriate cortices do have such neurons, we discuss the possibility that the extrastriate cortices play no role in guiding exogenous attention so that they can be devoted to other functions like visual decoding and endogenous attention.     

Preparatory effects of distractor suppression: evidence from visual cortex

Spatial selective attention is the mechanism that facilitates the selection of relevant information over irrelevant information in the visual field. The current study investigated whether foreknowledge of the presence or absence of distractors surrounding an impending target stimulus results in preparatory changes in visual cortex. We cued the location of the target and the presence or absence of distractors surrounding the target while changes in blood oxygen level dependent (BOLD) signals were measured. In line with prior work, we found that top-down spatial attention resulted in an increased contralateral BOLD response, evoked by the cue throughout early visual cortex (areas V1, V2 and V3). In addition, cues indicating distractor presence evoked a substantial increase in the magnitude of the BOLD signal in visual area V3, but not in V2 or V1. This study shows that prior knowledge concerning the presence of a distractor results in enhanced attentional modulation of visual cortex, in visual areas where neuronal receptive fields are large enough to encompass both targets and distractors. We interpret these findings as evidence that top-down attentional control processes include active preparatory suppression mechanisms for irrelevant, distracting information in the visual scene.     

Saliency changes appearance

Numerous studies have suggested that the deployment of attention is linked to saliency. In contrast, very little is known about how salient objects are perceived. To probe the perception of salient elements, observers compared two horizontally aligned stimuli in an array of eight elements. One of them was salient because of its orientation or direction of motion. We observed that the perceived luminance contrast or color saturation of the salient element increased: the salient stimulus looked even more salient. We explored the possibility that changes in appearance were caused by attention. We chose an event-related potential indexing attentional selection, the N2pc, to answer this question. The absence of an N2pc to the salient object provides preliminary evidence against involuntary attentional capture by the salient element. We suggest that signals from a master saliency map flow back into individual feature maps. These signals boost the perceived feature contrast of salient objects, even on perceptual dimensions different from the one that initially defined saliency.     

Properties of V1 neurons tuned to conjunctions of visual features: application of the V1 saliency hypothesis to visual search behavior

From a computational theory of V1, we formulate an optimization problem to investigate neural properties in the primary visual cortex (V1) from human reaction times (RTs) in visual search. The theory is the V1 saliency hypothesis that the bottom-up saliency of any visual location is represented by the highest V1 response to it relative to the background responses. The neural properties probed are those associated with the less known V1 neurons tuned simultaneously or conjunctively in two feature dimensions. The visual search is to find a target bar unique in color (C), orientation (O), motion direction (M), or redundantly in combinations of these features (e.g., CO, MO, or CM) among uniform background bars. A feature singleton target is salient because its evoked V1 response largely escapes the iso-feature suppression on responses to the background bars. The responses of the conjunctively tuned cells are manifested in the shortening of the RT for a redundant feature target (e.g., a CO target) from that predicted by a race between the RTs for the two corresponding single feature targets (e.g., C and O targets). Our investigation enables the following testable predictions. Contextual suppression on the response of a CO-tuned or MO-tuned conjunctive cell is weaker when the contextual inputs differ from the direct inputs in both feature dimensions, rather than just one. Additionally, CO-tuned cells and MO-tuned cells are often more active than the single feature tuned cells in response to the redundant feature targets, and this occurs more frequently for the MO-tuned cells such that the MO-tuned cells are no less likely than either the M-tuned or O-tuned neurons to be the most responsive neuron to dictate saliency for an MO target.     

Visual attention: the where,what, how and why of saliency

Attention influences the processing of visual information even in the earliest areas of primate visual cortex. There is converging evidence that the interaction of bottom-up sensory information and top-down attentional influences creates an integrated saliency map, that is, a topographic representation of relative stimulus strength and behavioral relevance across visual space. This map appears to be distributed across areas of the visual cortex, and is closely linked to the oculomotor system that controls eye movements and orients the gaze to locations in the visual scene characterized by a high salience.     

Saccade generation by the frontal eye fields in rhesus monkeys is separable from visual detection and bottom-up attention shift

The frontal eye fields (FEF), originally identified as an oculomotor cortex, have also been implicated in perceptual functions, such as constructing a visual saliency map and shifting visual attention. Further dissecting the area's role in the transformation from visual input to oculomotor command has been difficult because of spatial confounding between stimuli and responses and consequently between intermediate cognitive processes, such as attention shift and saccade preparation. Here we developed two tasks in which the visual stimulus and the saccade response were dissociated in space (the extended memory-guided saccade task), and bottom-up attention shift and saccade target selection were independent (the four-alternative delayed saccade task). Reversible inactivation of the FEF in rhesus monkeys disrupted, as expected, contralateral memory-guided saccades, but visual detection was demonstrated to be intact at the same field. Moreover, saccade behavior was impaired when a bottom-up shift of attention was not a prerequisite for saccade target selection, indicating that the inactivation effect was independent of the previously reported dysfunctions in bottom-up attention control. These findings underscore the motor aspect of the area's functions, especially in situations where saccades are generated by internal cognitive processes, including visual short-term memory and long-term associative memory.     

An embedded saliency map estimator scheme: application to video encoding

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.     

Modulation of activity in human visual area V1 during memory masking

Neurons in the primary visual cortex, V1, are specialized for the processing of elemental features of the visual stimulus, such as orientation and spatial frequency. Recent fMRI evidence suggest that V1 neurons are also recruited in visual perceptual memory; a number of studies using multi-voxel pattern analysis have successfully decoded stimulus-specific information from V1 activity patterns during the delay phase in memory tasks. However, consistent fMRI signal modulations reflecting the memory process have not yet been demonstrated. Here, we report evidence, from three subjects, that the low V1 BOLD activity during retention of low-level visual features is caused by competing interactions between neural populations coding for different values along the spectrum of the dimension remembered. We applied a memory masking paradigm in which the memory representation of a masker stimulus interferes with a delayed spatial frequency discrimination task when its frequency differs from the discriminanda with ±1 octave and found that impaired behavioral performance due to masking is reflected in weaker V1 BOLD signals. This cross-channel inhibition in V1 only occurs with retinotopic overlap between the masker and the sample stimulus of the discrimination task. The results suggest that memory for spatial frequency is a local process in the retinotopically organized visual cortex.     

Decoding successive computational stages of saliency processing

An important requirement for vision is to identify interesting and relevant regions of the environment for further processing. Some models assume that salient locations from a visual scene are encoded in a dedicated spatial saliency map [1, 2]. Then, a winner-take-all (WTA) mechanism [1, 2] is often believed to threshold the graded saliency representation and identify the most salient position in the visual field. Here we aimed to assess whether neural representations of graded saliency and the subsequent WTA mechanism can be dissociated. We presented images of natural scenes while subjects were in a scanner performing a demanding fixation task, and thus their attention was directed away. Signals in early visual cortex and posterior intraparietal sulcus (IPS) correlated with graded saliency as defined by a computational saliency model. Multivariate pattern classification [3, 4] revealed that the most salient position in the visual field was encoded in anterior IPS and frontal eye fields (FEF), thus reflecting a potential WTA stage. Our results thus confirm that graded saliency and WTA-thresholded saliency are encoded in distinct neural structures. This could provide the neural representation required for rapid and automatic orientation toward salient events in natural environments.     

Stimulus saliency modulates pre-attentive processing speed in human visual cortex

The notion of a saliency-based processing architecture [1] underlying human vision is central to a number of current theories of visual selective attention [e.g., 2]. On this view, focal-attention is guided by an overall-saliency map of the scene, which integrates (sums) signals from pre-attentive sensory feature-contrast computations (e.g., for color, motion, etc.). By linking the Posterior Contralateral Negativity (PCN) component to reaction time (RT) performance, we tested one specific prediction of such salience summation models: expedited shifts of focal-attention to targets with low, as compared to high, target-distracter similarity. For two feature-dimensions (color and orientation), we observed decreasing RTs with increasing target saliency. Importantly, this pattern was systematically mirrored by the timing, as well as amplitude, of the PCN. This pattern demonstrates that visual saliency is a key determinant of the time it takes for focal-attention to be engaged onto the target item, even when it is just a feature singleton.     

Synaptic integration by V1 neurons depends on location within the orientation map

Neurons in the primary visual cortex (V1) are organized into an orientation map consisting of orientation domains arranged radially around "pinwheel centers" at which the representations of all orientations converge. We have combined optical imaging of intrinsic signals with intracellular recordings to estimate the subthreshold inputs and spike outputs of neurons located near pinwheel centers or in orientation domains. We find that neurons near pinwheel centers have subthreshold responses to all stimulus orientations but spike responses to only a narrow range of orientations. Across the map, the selectivity of inputs covaries with the selectivity of orientations in the local cortical network, while the selectivity of spike outputs does not. Thus, the input-output transformation performed by V1 neurons is powerfully influenced by the local structure of the orientation map.     

Distribution of attention modulates salience signals in early visual cortex

Previous research has shown that the extent to which people spread attention across the visual field plays a crucial role in visual selection and the occurrence of bottom-up driven attentional capture. Consistent with previous findings, we show that when attention was diffusely distributed across the visual field while searching for a shape singleton, an irrelevant salient color singleton captured attention. However, while using the very same displays and task, no capture was observed when observers initially focused their attention at the center of the display. Using event-related fMRI, we examined the modulation of retinotopic activity related to attentional capture in early visual areas. Because the sensory display characteristics were identical in both conditions, we were able to isolate the brain activity associated with exogenous attentional capture. The results show that spreading of attention leads to increased bottom-up exogenous capture and increased activity in visual area V3 but not in V2 and V1.     

Exploring BOLD Changes during Spatial Attention in Non-Stimulated Visual Cortex

Blood oxygen level-dependent (BOLD) responses were measured in parts of primary visual cortex that represented unstimulated visual field regions at different distances from a stimulated central target location. The composition of the visual scene varied by the presence or absence of additional peripheral distracter stimuli. Bottom-up effects were assessed by comparing peripheral activity during central stimulation vs. no stimulation. Top-down effects were assessed by comparing active vs. passive conditions. In passive conditions subjects simply watched the central letter stimuli and in active conditions they had to report occurrence of pre-defined targets in a rapid serial letter stream. Onset of the central letter stream enhanced activity in V1 representations of the stimulated region. Within representations of the periphery activation decreased and finally turned into deactivation with increasing distance from the stimulated location. This pattern was most pronounced in the active conditions and during the presence of peripheral stimuli. Active search for a target did not lead to additional enhancement at areas representing the attentional focus but to a stronger deactivation in the vicinity. Suppressed neuronal activity was also found in the non distracter condition suggesting a top-down attention driven effect. Our observations suggest that BOLD signal decreases in primary visual cortex are modulated by bottom-up sensory-driven factors such as the presence of distracters in the visual field as well as by top-down attentional processes.     

Saliency computation via whitened frequency band selection

Many saliency computational models have been proposed to simulate bottom-up visual attention mechanism of human visual system. However, most of them only deal with certain kinds of images or aim at specific applications. In fact, human beings have the ability to correctly select attentive focuses of objects with arbitrary sizes within any scenes. This paper proposes a new bottom-up computational model from the perspective of frequency domain based on the biological discovery of non-Classical Receptive Field (nCRF) in the retina. A saliency map can be obtained according to the idea of Extended Classical Receptive Field. The model is composed of three major steps: firstly decompose the input image into several feature maps representing different frequency bands that cover the whole frequency domain by utilizing Gabor wavelet. Secondly, whiten the feature maps to highlight the embedded saliency information. Thirdly, select some optimal maps, simulating the response of receptive field especially nCRF, to generate the saliency map. Experimental results show that the proposed algorithm is able to work with stable effect and outstanding performance in a variety of situations as human beings do and is adaptive to both psychological patterns and natural images. Beyond that, biological plausibility of nCRF and Gabor wavelet transform make this approach reliable.     

Stability of cortical responses and the statistics of natural scenes.

The primary visual cortex (V1) of higher mammals contains maps of stimulus features; how these maps influence vision remains unknown. We have examined the functional significance of an asymmetry in the orientation map in cat V1, i.e., the fact that a larger area of V1 is preferentially activated by vertical and horizontal contours than by contours at oblique orientations. Despite the fact that neurons tuned to cardinal and oblique orientations have indistinguishable tuning characteristics, cardinal neurons remain more stable in their response properties after selective perturbation induced by adaptation. Similarly, human observers report different adaptation-induced changes in orientation tuning between cardinal and oblique axes. We suggest that the larger cortical area devoted to cardinal orientations imposes stability on the processing of cardinal contours during visual perception, by retaining invariant cortical responses along cardinal axes.     

Visual Saliency Models for Text Detection in Real World

This paper evaluates the degree of saliency of texts in natural scenes using visual saliency models. A large scale scene image database with pixel level ground truth is created for this purpose. Using this scene image database and five state-of-the-art models, visual saliency maps that represent the degree of saliency of the objects are calculated. The receiver operating characteristic curve is employed in order to evaluate the saliency of scene texts, which is calculated by visual saliency models. A visualization of the distribution of scene texts and non-texts in the space constructed by three kinds of saliency maps, which are calculated using Itti''s visual saliency model with intensity, color and orientation features, is given. This visualization of distribution indicates that text characters are more salient than their non-text neighbors, and can be captured from the background. Therefore, scene texts can be extracted from the scene images. With this in mind, a new visual saliency architecture, named hierarchical visual saliency model, is proposed. Hierarchical visual saliency model is based on Itti''s model and consists of two stages. In the first stage, Itti''s model is used to calculate the saliency map, and Otsu''s global thresholding algorithm is applied to extract the salient region that we are interested in. In the second stage, Itti''s model is applied to the salient region to calculate the final saliency map. An experimental evaluation demonstrates that the proposed model outperforms Itti''s model in terms of captured scene texts.     

Brain activity accompanying perception of implied motion in abstract paintings

Early 20th century artists including Duchamp and Balla tried to portray moving objects on a static canvas by superimposing objects in successive portrayals of an action. We investigated whether implied motion in those paintings is associated with activation of motion-sensitive area MT+. In Experiment 1, we found that observers rated these kinds of paintings higher in portraying motion than they did other abstract paintings in which motion is not intended. We also found that observers who had previously experienced abstract paintings with implied motion tended to give higher motion ratings to that class of paintings. In Experiment 2, we used functional magnetic resonance imaging (fMRI) to measure brain activity of observers while viewing abstract paintings receiving the highest and the lowest motion rating scores in Experiment 1. We found MT+, but not primary visual cortex (V1), showed greater BOLD responses to abstract paintings with implied motion than to abstract paintings with little motion impression, but only in observers with prior experience viewing those kinds of paintings. These results imply that the neural machinery ordinarily engaged during perception of real visual motion is activated when people view paintings explicitly designed to convey a sense of visual motion. Experience, however, is necessary to achieve this sense of motion.     

Visually Driven Activation in Macaque Areas V2 and V3 without Input from the Primary Visual Cortex

Creating focal lesions in primary visual cortex (V1) provides an opportunity to study the role of extra-geniculo-striate pathways for activating extrastriate visual cortex. Previous studies have shown that more than 95% of neurons in macaque area V2 and V3 stop firing after reversibly cooling V1 [1], [2], [3]. However, no studies on long term recovery in areas V2, V3 following permanent V1 lesions have been reported in the macaque. Here we use macaque fMRI to study area V2, V3 activity patterns from 1 to 22 months after lesioning area V1. We find that visually driven BOLD responses persist inside the V1-lesion projection zones (LPZ) of areas V2 and V3, but are reduced in strength by ∼70%, on average, compared to pre-lesion levels. Monitoring the LPZ activity over time starting one month following the V1 lesion did not reveal systematic changes in BOLD signal amplitude. Surprisingly, the retinotopic organization inside the LPZ of areas V2, V3 remained similar to that of the non-lesioned hemisphere, suggesting that LPZ activation in V2, V3 is not the result of input arising from nearby (non-lesioned) V1 cortex. Electrophysiology recordings of multi-unit activity corroborated the BOLD observations: visually driven multi-unit responses could be elicited inside the V2 LPZ, even when the visual stimulus was entirely contained within the scotoma induced by the V1 lesion. Restricting the stimulus to the intact visual hemi-field produced no significant BOLD modulation inside the V2, V3 LPZs. We conclude that the observed activity patterns are largely mediated by parallel, V1-bypassing, subcortical pathways that can activate areas V2 and V3 in the absence of V1 input. Such pathways may contribute to the behavioral phenomenon of blindsight.