Saliency computation via whitened frequency band selection

Many saliency computational models have been proposed to simulate bottom-up visual attention mechanism of human visual system. However, most of them only deal with certain kinds of images or aim at specific applications. In fact, human beings have the ability to correctly select attentive focuses of objects with arbitrary sizes within any scenes. This paper proposes a new bottom-up computational model from the perspective of frequency domain based on the biological discovery of non-Classical Receptive Field (nCRF) in the retina. A saliency map can be obtained according to the idea of Extended Classical Receptive Field. The model is composed of three major steps: firstly decompose the input image into several feature maps representing different frequency bands that cover the whole frequency domain by utilizing Gabor wavelet. Secondly, whiten the feature maps to highlight the embedded saliency information. Thirdly, select some optimal maps, simulating the response of receptive field especially nCRF, to generate the saliency map. Experimental results show that the proposed algorithm is able to work with stable effect and outstanding performance in a variety of situations as human beings do and is adaptive to both psychological patterns and natural images. Beyond that, biological plausibility of nCRF and Gabor wavelet transform make this approach reliable.     

Visual attention: the where,what, how and why of saliency

Attention influences the processing of visual information even in the earliest areas of primate visual cortex. There is converging evidence that the interaction of bottom-up sensory information and top-down attentional influences creates an integrated saliency map, that is, a topographic representation of relative stimulus strength and behavioral relevance across visual space. This map appears to be distributed across areas of the visual cortex, and is closely linked to the oculomotor system that controls eye movements and orients the gaze to locations in the visual scene characterized by a high salience.     

Spontaneous symmetry breaking in self–organizing neural fields

We extend the theory of self-organizing neural fields in order to analyze the joint emergence of topography and feature selectivity in primary visual cortex through spontaneous symmetry breaking. We first show how a binocular one-dimensional topographic map can undergo a pattern forming instability that breaks the underlying symmetry between left and right eyes. This leads to the spatial segregation of eye specific activity bumps consistent with the emergence of ocular dominance columns. We then show how a 2-dimensional isotropic topographic map can undergo a pattern forming instability that breaks the underlying rotation symmetry. This leads to the formation of elongated activity bumps consistent with the emergence of orientation preference columns. A particularly interesting property of the latter symmetry breaking mechanism is that the linear equations describing the growth of the orientation columns exhibits a rotational shift-twist symmetry, in which there is a coupling between orientation and topography. Such coupling has been found in experimentally generated orientation preference maps     

Neural activities in v1 create a bottom-up saliency map

The bottom-up contribution to the allocation of exogenous attention is a saliency map, whose neural substrate is hard to identify because of possible contamination by top-down signals. We obviated this possibility using stimuli that observers could not perceive, but that nevertheless, through orientation contrast between foreground and background regions, attracted attention to improve a localized visual discrimination. When orientation contrast increased, so did the degree of attraction, and two physiological measures: the amplitude of the earliest (C1) component of the ERP, which is associated with primary visual cortex, and fMRI BOLD signals in areas V1-V4 (but not the intraparietal sulcus). Significantly, across observers, the degree of attraction correlated with the C1 amplitude and just the V1 BOLD signal. These findings strongly support the proposal that a bottom-up saliency map is created in V1, challenging the dominant view that the saliency map is generated in the parietal cortex.     

Psychophysical tests of the hypothesis of a bottom-up saliency map in primary visual cortex

A unique vertical bar among horizontal bars is salient and pops out perceptually. Physiological data have suggested that mechanisms in the primary visual cortex (V1) contribute to the high saliency of such a unique basic feature, but indicated little regarding whether V1 plays an essential or peripheral role in input-driven or bottom-up saliency. Meanwhile, a biologically based V1 model has suggested that V1 mechanisms can also explain bottom-up saliencies beyond the pop-out of basic features, such as the low saliency of a unique conjunction feature such as a red vertical bar among red horizontal and green vertical bars, under the hypothesis that the bottom-up saliency at any location is signaled by the activity of the most active cell responding to it regardless of the cell's preferred features such as color and orientation. The model can account for phenomena such as the difficulties in conjunction feature search, asymmetries in visual search, and how background irregularities affect ease of search. In this paper, we report nontrivial predictions from the V1 saliency hypothesis, and their psychophysical tests and confirmations. The prediction that most clearly distinguishes the V1 saliency hypothesis from other models is that task-irrelevant features could interfere in visual search or segmentation tasks which rely significantly on bottom-up saliency. For instance, irrelevant colors can interfere in an orientation-based task, and the presence of horizontal and vertical bars can impair performance in a task based on oblique bars. Furthermore, properties of the intracortical interactions and neural selectivities in V1 predict specific emergent phenomena associated with visual grouping. Our findings support the idea that a bottom-up saliency map can be at a lower visual area than traditionally expected, with implications for top-down selection mechanisms.     

Training Top-Down Attention Improves Performance on a Triple-Conjunction Search Task

Training has been shown to improve perceptual performance on limited sets of stimuli. However, whether training can generally improve top-down biasing of visual search in a target-nonspecific manner remains unknown. We trained subjects over ten days on a visual search task, challenging them with a novel target (top-down goal) on every trial, while bottom-up uncertainty (distribution of distractors) remained constant. We analyzed the changes in saccade statistics and visual behavior over the course of training by recording eye movements as subjects performed the task. Subjects became experts at this task, with twofold increased performance, decreased fixation duration, and stronger tendency to guide gaze toward items with color and spatial frequency (but not necessarily orientation) that resembled the target, suggesting improved general top-down biasing of search.     

Goal-related activity in V4 during free viewing visual search. Evidence for a ventral stream visual salience map

Natural exploration of complex visual scenes depends on saccadic eye movements toward important locations. Saccade targeting is thought to be mediated by a retinotopic map that represents the locations of salient features. In this report, we demonstrate that extrastriate ventral area V4 contains a retinotopic salience map that guides exploratory eye movements during a naturalistic free viewing visual search task. In more than half of recorded cells, visually driven activity is enhanced prior to saccades that move the fovea toward the location previously occupied by a neuron's spatial receptive field. This correlation suggests that bottom-up processing in V4 influences the oculomotor planning process. Half of the neurons also exhibit top-down modulation of visual responses that depends on search target identity but not visual stimulation. Convergence of bottom-up and top-down processing streams in area V4 results in an adaptive, dynamic map of salience that guides oculomotor planning during natural vision.     

Predicting the eye fixation locations in the gray scale images in the visual scenes with different semantic contents

In recent years, there has been considerable interest in visual attention models (saliency map of visual attention). These models can be used to predict eye fixation locations, and thus will have many applications in various fields which leads to obtain better performance in machine vision systems. Most of these models need to be improved because they are based on bottom-up computation that does not consider top-down image semantic contents and often does not match actual eye fixation locations. In this study, we recorded the eye movements (i.e., fixations) of fourteen individuals who viewed images which consist natural (e.g., landscape, animal) and man-made (e.g., building, vehicles) scenes. We extracted the fixation locations of eye movements in two image categories. After extraction of the fixation areas (a patch around each fixation location), characteristics of these areas were evaluated as compared to non-fixation areas. The extracted features in each patch included the orientation and spatial frequency. After feature extraction phase, different statistical classifiers were trained for prediction of eye fixation locations by these features. This study connects eye-tracking results to automatic prediction of saliency regions of the images. The results showed that it is possible to predict the eye fixation locations by using of the image patches around subjects’ fixation points.     

Functional topography of connections linking mirror-symmetric maps in the mouse olfactory bulb

In rodents, each main olfactory bulb contains two mirror-symmetric glomerular maps, a feature not found in the initial topographic maps of other sensory systems. Targeting tracer injections to identified glomeruli revealed that isofunctional odor columns-translaminar assemblies connected to a given glomerulus-were specifically and reciprocally interconnected through a mutually inhibitory circuit with exquisite topographic specificity. Thus, instead of containing two mirror-symmetric maps, we propose that the olfactory bulb contains a single integrated map in which isofunctional odor columns are connected through an intrabulbar link, analogous to the specific horizontal connections linking iso-orientation columns in primary visual cortex.     

The coordinated mapping of visual space and response features in visual cortex

Whether general principles can explain the layouts of cortical maps remains unresolved. In primary visual cortex of ferret, the relationships between the maps of visual space and response features are predicted by a "dimension-reduction" model. The representation of visual space is anisotropic, with the elevation and azimuth axes having different magnification. This anisotropy is reflected in the orientation, ocular dominance, and spatial frequency domains, which are elongated such that their directions of rapid change, or high-gradient axes, are orthogonal to the high-gradient axis of the visual map. The feature maps are also strongly interdependent-their high-gradient regions avoid one another and intersect orthogonally where essential, so that overlap is minimized. Our results demonstrate a clear influence of the visual map on each feature map. In turn, the local representation of visual space is smooth, as predicted when many features are mapped within a cortical area.     

Top-Down but Not Bottom-Up Visual Scanning is Affected in Hereditary Pure Cerebellar Ataxia

The aim of this study was to clarify the nature of visual processing deficits caused by cerebellar disorders. We studied the performance of two types of visual search (top-down visual scanning and bottom-up visual scanning) in 18 patients with pure cerebellar types of spinocerebellar degeneration (SCA6: 11; SCA31: 7). The gaze fixation position was recorded with an eye-tracking device while the subjects performed two visual search tasks in which they looked for a target Landolt figure among distractors. In the serial search task, the target was similar to the distractors and the subject had to search for the target by processing each item with top-down visual scanning. In the pop-out search task, the target and distractor were clearly discernible and the visual salience of the target allowed the subjects to detect it by bottom-up visual scanning. The saliency maps clearly showed that the serial search task required top-down visual attention and the pop-out search task required bottom-up visual attention. In the serial search task, the search time to detect the target was significantly longer in SCA patients than in normal subjects, whereas the search time in the pop-out search task was comparable between the two groups. These findings suggested that SCA patients cannot efficiently scan a target using a top-down attentional process, whereas scanning with a bottom-up attentional process is not affected. In the serial search task, the amplitude of saccades was significantly smaller in SCA patients than in normal subjects. The variability of saccade amplitude (saccadic dysmetria), number of re-fixations, and unstable fixation (nystagmus) were larger in SCA patients than in normal subjects, accounting for a substantial proportion of scattered fixations around the items. Saccadic dysmetria, re-fixation, and nystagmus may play important roles in the impaired top-down visual scanning in SCA, hampering precise visual processing of individual items.     

Feature-based attention: it is all bottom-up priming

Feature-based attention (FBA) enhances the representation of image characteristics throughout the visual field, a mechanism that is particularly useful when searching for a specific stimulus feature. Even though most theories of visual search implicitly or explicitly assume that FBA is under top-down control, we argue that the role of top-down processing in FBA may be limited. Our review of the literature indicates that all behavioural and neuro-imaging studies investigating FBA suffer from the shortcoming that they cannot rule out an effect of priming. The mere attending to a feature enhances the mandatory processing of that feature across the visual field, an effect that is likely to occur in an automatic, bottom-up way. Studies that have investigated the feasibility of FBA by means of cueing paradigms suggest that the role of top-down processing in FBA is limited (e.g. prepare for red). Instead, the actual processing of the stimulus is needed to cause the mandatory tuning of responses throughout the visual field. We conclude that it is likely that all FBA effects reported previously are the result of bottom-up priming.     

Modeling feature-based attention as an active top-down inference process

Vision is a crucial sensor. It provides a very rich collection of information about our environment. The difficulty in vision arises, since this information is not obvious in the image, it has to be constructed. Wheres earlier approaches have favored a bottom-up approach, which maps the image onto an internal representation of the world, more recent approaches search for alternatives and develop frameworks which make use of top-down connections. In these approaches vision is inherently a constructive process which makes use of a priory information. Following this line of research, a model of primate object perception is presented and used to simulate an object detection task in natural scenes. The model predicts that early responses in extrastriate visual areas are modulated by the visual goal.     

Modeling feature-based attention as an active top-down inference process

Vision is a crucial sensor. It provides a very rich collection of information about our environment. The difficulty in vision arises, since this information is not obvious in the image, it has to be constructed. Wheres earlier approaches have favored a bottom-up approach, which maps the image onto an internal representation of the world, more recent approaches search for alternatives and develop frameworks which make use of top-down connections. In these approaches vision is inherently a constructive process which makes use of a priory information. Following this line of research, a model of primate object perception is presented and used to simulate an object detection task in natural scenes. The model predicts that early responses in extrastriate visual areas are modulated by the visual goal.     

The Primary Visual Cortex Is Differentially Modulated by Stimulus-Driven and Top-Down Attention

Selective attention can be focused either volitionally, by top-down signals derived from task demands, or automatically, by bottom-up signals from salient stimuli. Because the brain mechanisms that underlie these two attention processes are poorly understood, we recorded local field potentials (LFPs) from primary visual cortical areas of cats as they performed stimulus-driven and anticipatory discrimination tasks. Consistent with our previous observations, in both tasks, we found enhanced beta activity, which we have postulated may serve as an attention carrier. We characterized the functional organization of task-related beta activity by (i) cortical responses (EPs) evoked by electrical stimulation of the optic chiasm and (ii) intracortical LFP correlations. During the anticipatory task, peripheral stimulation that was preceded by high-amplitude beta oscillations evoked large-amplitude EPs compared with EPs that followed low-amplitude beta. In contrast, during the stimulus-driven task, cortical EPs preceded by high-amplitude beta oscillations were, on average, smaller than those preceded by low-amplitude beta. Analysis of the correlations between the different recording sites revealed that beta activation maps were heterogeneous during the bottom-up task and homogeneous for the top-down task. We conclude that bottom-up attention activates cortical visual areas in a mosaic-like pattern, whereas top-down attentional modulation results in spatially homogeneous excitation.     

The Caenorhabditis elegans Six/sine oculis class homeobox gene ceh-32 is required for head morphogenesis

Repulsion plays a fundamental role in the establishment of a topographic map of the chick retinotectal projections. This has been highlighted by studies demonstrating the role of opposing gradients of the EphA3 receptor tyrosine kinase on retinal axons and two of its ligands, ephrin-A2 and ephrin-A5, in the tectum. We have analyzed the distribution of these two ephrins in other retinorecipient structures in the chick diencephalon and mesencephalon during the period when visual connections are being established. We have found that both ephrin-A2 and ephrin-A5 and their receptors EphA4 and EphA7 are expressed in gradients whose orientation is consistent with the topography of the nasotemporal axis of the respective retinofugal projections. In addition, their distribution suggests that receptor-ligand interactions may be involved in the organization of connections between the different primary visual centers and, thus, in the topographic organization of secondary visual projections. Interestingly, where projections lack a clear topographic representation, a uniform expression of the Eph-ephrin molecules was observed. Finally, we also show that a similar patterning mechanism may be implicated in the transfer of visual information to the telencephalon. These results suggest a conserved function for EphA receptors and their ligands in the elaboration of topographic maps at multiple levels of the visual pathway.     

How does the cerebral cortex work? Learning, attention, and grouping by the laminar circuits of visual cortex

The organization of neocortex into layers is one of its most salient anatomical features. These layers include circuits that form functional columns in cortical maps. A major unsolved problem concerns how bottom-up, top-down, and horizontal interactions are organized within cortical layers to generate adaptive behaviors. This article models how these interactions help visual cortex to realize: (i) the binding process whereby cortex groups distributed data into coherent object representations; (ii) the attentional process whereby cortex selectively processes important events; and (iii) the developmental and learning processes whereby cortex shapes its circuits to match environmental constraints. New computational ideas about feedback systems suggest how neocortex develops and learns in a stable way, and why top-down attention requires converging bottom-up inputs to fully activate cortical cells, whereas perceptual groupings do not.     

Coordinated Optimization of Visual Cortical Maps (I) Symmetry-based Analysis

In the primary visual cortex of primates and carnivores, functional architecture can be characterized by maps of various stimulus features such as orientation preference (OP), ocular dominance (OD), and spatial frequency. It is a long-standing question in theoretical neuroscience whether the observed maps should be interpreted as optima of a specific energy functional that summarizes the design principles of cortical functional architecture. A rigorous evaluation of this optimization hypothesis is particularly demanded by recent evidence that the functional architecture of orientation columns precisely follows species invariant quantitative laws. Because it would be desirable to infer the form of such an optimization principle from the biological data, the optimization approach to explain cortical functional architecture raises the following questions: i) What are the genuine ground states of candidate energy functionals and how can they be calculated with precision and rigor? ii) How do differences in candidate optimization principles impact on the predicted map structure and conversely what can be learned about a hypothetical underlying optimization principle from observations on map structure? iii) Is there a way to analyze the coordinated organization of cortical maps predicted by optimization principles in general? To answer these questions we developed a general dynamical systems approach to the combined optimization of visual cortical maps of OP and another scalar feature such as OD or spatial frequency preference. From basic symmetry assumptions we obtain a comprehensive phenomenological classification of possible inter-map coupling energies and examine representative examples. We show that each individual coupling energy leads to a different class of OP solutions with different correlations among the maps such that inferences about the optimization principle from map layout appear viable. We systematically assess whether quantitative laws resembling experimental observations can result from the coordinated optimization of orientation columns with other feature maps.     

Creating abstract topographic representations: Implications for coding,learning and reasoning

Topographic maps are a fundamental and ubiquitous feature of the sensory and motor regions of the brain. There is less evidence for the existence of conventional topographic maps in associational areas of the brain such as the prefrontal cortex and parietal cortex. The existence of topographically arranged anatomical projections is far more widespread and occurs in associational regions of the brain as well as sensory and motor regions: this points to a more widespread existence of topographically organised maps within associational cortex than currently recognised. Indeed, there is increasing evidence that abstract topographic representations may also occur in these regions. For example, a topographic mnemonic map of visual space has been described in the dorsolateral prefrontal cortex and topographically arranged visuospatial attentional signals have been described in parietal association cortex. This article explores how abstract representations might be extracted from sensory topographic representations and subsequently code abstract information. Finally a simple model is presented that shows how abstract topographic representations could be integrated with other information within the brain to solve problems or form abstract associations. The model uses correlative firing to detect associations between different types of stimuli. It is flexible because it can produce correlations between information represented in a topographic or non-topographic coordinate system. It is proposed that a similar process could be used in high-level cognitive operations such as learning and reasoning.     

Spatiotemporal Dynamics of High-Gamma Activities during a 3-Stimulus Visual Oddball Task

Although many studies have investigated the neural basis of top-down and bottom-up attention, it still requires refinement in both temporal and spatial terms. We used magnetoencephalography to investigate the spatiotemporal dynamics of high-gamma (52–100 Hz) activities during top-down and bottom-up visual attentional processes, aiming to extend the findings from functional magnetic resonance imaging and event-related potential studies. Fourteen participants performed a 3-stimulus visual oddball task, in which both infrequent non-target and target stimuli were presented. We identified high-gamma event-related synchronization in the left middle frontal gyrus, the left intraparietal sulcus, the left thalamus, and the visual areas in different time windows for the target and non-target conditions. We also found elevated imaginary coherence between the left intraparietal sulcus and the right middle frontal gyrus in the high-gamma band from 300 to 400 ms in the target condition, and between the left thalamus and the left middle frontal gyrus in theta band from 150 to 450 ms. In addition, the strength of high-gamma imaginary coherence between the left middle frontal gyrus and left intraparietal sulcus, between the left middle frontal gyrus and the right middle frontal gyrus, and the high-gamma power in the left thalamus predicted inter-subject variation in target detection response time. This source-level electrophysiological evidence enriches our understanding of bi-directional attention processes: stimulus-driven bottom-up attention orientation to a salient, but irrelevant stimulus; and top-down allocation of attentional resources to stimulus evaluation.