Contour integration and cortical processing.

Our understanding of visual processing in general, and contour integration in particular, has undergone great change over the last 10 years. There is now an accumulation of psychophysical and neurophysiological evidence that the outputs of cells with conjoint orientation preference and spatial position are integrated in the process of explication of rudimentary contours. Recent neuroanatomical and neurophysiological results suggest that this process takes place at the cortical level V1. The code for contour integration may be a temporal one in that it may only manifest itself in the latter part of the spike train as a result of feedback and lateral interactions. Here we review some of the properties of contour integration from a psychophysical perspective and we speculate on their underlying neurophysiological substrate.     

Cortical connections and early visual function: intra- and inter-columnar processing.

While it is widely assumed that the long-range horizontal connections in V1 are present to support contour integration, there has been only limited consideration of other possible relationships between anatomy and physiology (the horizontal connections) and visual function beyond contour integration. We introduce the possibility of other relationships directly from the perspective of computation and differential geometry by identifying orientation columns in visual physiology with the (unit) tangent bundle in differential geometry. This suggests abstracting early vision in a space that incorporates both position and orientation, from which we show that the physiology is capable of supporting a number of functional computations beyond contour integration, including texture-flow and shading-flow integration, as well as certain relationships between them. The geometric abstraction emphasizes the role of curvature, which necessitates a coupled investigation into how it might be estimated. The result is an elaboration of layer-to-layer interactions within an orientation column, with non-linearities possibly implemented by shunting inhibition. Finally, we show how the same computational framework naturally lends itself to solving stereo correspondence, with binocular tangents abstracting curves in space.     

Collinear interactions and contour integration

The visibility of a local target is influenced by the global configuration of the stimulus. Collinear configurations are a specific case in which facilitation or suppression of the target has been found to be dependent on the contrast threshold of the target. The role of collinear interactions in perceptual grouping, especially in contour integration, is still controversial. In the current study, the role of collinear interactions in noise was investigated using experimental conditions similar to those utilized in studies of contour integration. The contrast detection paradigm in the presence of similar Gabor elements presented in the background was used. The results show that contrast detection threshold of the target alone is increased (suppression) when it is embedded in randomly oriented background elements. However, when the target is flanked by two collinear Gabor elements, the target is facilitated even at higher target contrast levels. Facilitation is not found for orthogonal configurations. The results suggest that the response to a local element in a contour is modified by lateral facilitative and suppressive inputs from elements comprising the smooth contour and randomly oriented background elements, respectively. Thus, detection of elements along a contour should be considered as integration of global neuronal activity rather than as the output of local and individual neurons.     

Computational modeling and exploration of contour integration for visual saliency

We propose a computational model of contour integration for visual saliency. The model uses biologically plausible devices to simulate how the representations of elements aligned collinearly along a contour in an image are enhanced. Our model adds such devices as a dopamine-like fast plasticity, local GABAergic inhibition and multi-scale processing of images. The fast plasticity addresses the problem of how neurons in visual cortex seem to be able to influence neurons they are not directly connected to, for instance, as observed in contour closure effect. Local GABAergic inhibition is used to control gain in the system without using global mechanisms which may be non-plausible given the limited reach of axonal arbors in visual cortex. The model is then used to explore not only its validity in real and artificial images, but to discover some of the mechanisms involved in processing of complex visual features such as junctions and end-stops as well as contours. We present evidence for the validity of our model in several phases, starting with local enhancement of only a few collinear elements. We then test our model on more complex contour integration images with a large number of Gabor elements. Sections of the model are also extracted and used to discover how the model might relate contour integration neurons to neurons that process end-stops and junctions. Finally, we present results from real world images. Results from the model suggest that it is a good current approximation of contour integration in human vision. As well, it suggests that contour integration mechanisms may be strongly related to mechanisms for detecting end-stops and junction points. Additionally, a contour integration mechanism may be involved in finding features for objects such as faces. This suggests that visual cortex may be more information efficient and that neural regions may have multiple roles.     

Optimality of human contour integration

For processing and segmenting visual scenes, the brain is required to combine a multitude of features and sensory channels. It is neither known if these complex tasks involve optimal integration of information, nor according to which objectives computations might be performed. Here, we investigate if optimal inference can explain contour integration in human subjects. We performed experiments where observers detected contours of curvilinearly aligned edge configurations embedded into randomly oriented distractors. The key feature of our framework is to use a generative process for creating the contours, for which it is possible to derive a class of ideal detection models. This allowed us to compare human detection for contours with different statistical properties to the corresponding ideal detection models for the same stimuli. We then subjected the detection models to realistic constraints and required them to reproduce human decisions for every stimulus as well as possible. By independently varying the four model parameters, we identify a single detection model which quantitatively captures all correlations of human decision behaviour for more than 2000 stimuli from 42 contour ensembles with greatly varying statistical properties. This model reveals specific interactions between edges closely matching independent findings from physiology and psychophysics. These interactions imply a statistics of contours for which edge stimuli are indeed optimally integrated by the visual system, with the objective of inferring the presence of contours in cluttered scenes. The recurrent algorithm of our model makes testable predictions about the temporal dynamics of neuronal populations engaged in contour integration, and it suggests a strong directionality of the underlying functional anatomy.     

Top-down modulation of lateral interactions in early vision: does attention affect integration of the whole or just perception of the parts?

Attention can modulate sensitivity to local stimuli in early vision. But, can attention also modulate integration of local stimuli into global visual patterns? We recently measured effects of attention on the phenomenon of lateral interactions between collinear elements, commonly thought to reflect long-range mechanisms in early visual cortex underlying contour integration. We showed improved detection of low-contrast central Gabor targets in the context of collinear flankers, but only when the collinear flankers were attended for a secondary task rather than ignored in favor of an orthogonal flanker pair. Here, we contrast two hypotheses for how attention might modulate flanker influences on the target: by changing just local sensitivity to the flankers themselves (flanker-modulation-only hypothesis), or by weighting integrative connections between flanker and target (connection-weighting hypothesis). Modeled on the known nonlinear dependence of target visibility on collinear flanker contrast, the first hypothesis predicts that an increase in physical flanker contrast should readily offset any reduction in their effective contrast when ignored, thus eliminating attentional modulation. Conversely, the second hypothesis predicts that attentional modulation should persist even for the highest flanker contrasts. Our results showed the latter outcome and indicated that attention modulates flanker-target integration, rather than just processing of local flanker elements.     

Mechanisms for spatial integration in visual detection: a model based on lateral interactions

Recent studies of visual detection show a configuration dependent weak improvement of thresholds with the number of targets, which corresponds to a fourth-root power law. We find this result to be inconsistent with probability summation models, and account for it by a model of 'physiological' integration that is based on excitatory lateral interactions in the visual cortex. The model explains several phenomena which are confirmed by the experimental data, such as the absence of spatial and temporal uncertainty effects, temporal summation curves, and facilitation by a pedestal in 2AFC tasks. The summation exponents are dependent on the strength of the lateral interactions, and on the distance and orientation relationship between the elements.     

Perceptual learning in Williams syndrome: looking beyond averages

Williams Syndrome is a genetically determined neurodevelopmental disorder characterized by an uneven cognitive profile and surprisingly large neurobehavioral differences among individuals. Previous studies have already shown different forms of memory deficiencies and learning difficulties in WS. Here we studied the capacity of WS subjects to improve their performance in a basic visual task. We employed a contour integration paradigm that addresses occipital visual function, and analyzed the initial (i.e. baseline) and after-learning performance of WS individuals. Instead of pooling the very inhomogeneous results of WS subjects together, we evaluated individual performance by expressing it in terms of the deviation from the average performance of the group of typically developing subjects of similar age. This approach helped us to reveal information about the possible origins of poor performance of WS subjects in contour integration. Although the majority of WS individuals showed both reduced baseline and reduced learning performance, individual analysis also revealed a dissociation between baseline and learning capacity in several WS subjects. In spite of impaired initial contour integration performance, some WS individuals presented learning capacity comparable to learning in the typically developing population, and vice versa, poor learning was also observed in subjects with high initial performance levels. These data indicate a dissociation between factors determining initial performance and perceptual learning.     

Contour saliency in primary visual cortex

Contour integration is an important intermediate stage of object recognition, in which line segments belonging to an object boundary are perceptually linked and segmented from complex backgrounds. Contextual influences observed in primary visual cortex (V1) suggest the involvement of V1 in contour integration. Here, we provide direct evidence that, in monkeys performing a contour detection task, there was a close correlation between the responses of V1 neurons and the perceptual saliency of contours. Receiver operating characteristic analysis showed that single neuronal responses encode the presence or absence of a contour as reliably as the animal's behavioral responses. We also show that the same visual contours elicited significantly weaker neuronal responses when they were not detected in the detection task, or when they were unattended. Our results demonstrate that contextual interactions in V1 play a pivotal role in contour integration and saliency.     

Globally consistent depth sorting of overlapping 2D surfaces in a model using local recurrent interactions

The human visual system utilizes depth information as a major cue to group together visual items constituting an object and to segregate them from items belonging to other objects in the visual scene. Depth information can be inferred from a variety of different visual cues, such as disparity, occlusions and perspective. Many of these cues provide only local and relative information about the depth of objects. For example, at occlusions, T-junctions indicate the local relative depth precedence of surface patches. However, in order to obtain a globally consistent interpretation of the depth relations between the surfaces and objects in a visual scene, a mechanism is necessary that globally propagates such local and relative information. We present a computational framework in which depth information derived from T-junctions is propagated along surface contours using local recurrent interactions between neighboring neurons. We demonstrate that within this framework a globally consistent depth sorting of overlapping surfaces can be obtained on the basis of local interactions. Unlike previous approaches in which locally restricted cell interactions could merely distinguish between two depths (figure and ground), our model can also represent several intermediate depth positions. Our approach is an extension of a previous model of recurrent V1–V2 interaction for contour processing and illusory contour formation. Based on the contour representation created by this model, a recursive scheme of local interactions subsequently achieves a globally consistent depth sorting of several overlapping surfaces. Within this framework, the induction of illusory contours by the model of recurrent V1–V2 interaction gives rise to the figure-ground segmentation of illusory figures such as a Kanizsa square.     

Model cortical association fields account for the time course and dependence on target complexity of human contour perception

Can lateral connectivity in the primary visual cortex account for the time dependence and intrinsic task difficulty of human contour detection? To answer this question, we created a synthetic image set that prevents sole reliance on either low-level visual features or high-level context for the detection of target objects. Rendered images consist of smoothly varying, globally aligned contour fragments (amoebas) distributed among groups of randomly rotated fragments (clutter). The time course and accuracy of amoeba detection by humans was measured using a two-alternative forced choice protocol with self-reported confidence and variable image presentation time (20-200 ms), followed by an image mask optimized so as to interrupt visual processing. Measured psychometric functions were well fit by sigmoidal functions with exponential time constants of 30-91 ms, depending on amoeba complexity. Key aspects of the psychophysical experiments were accounted for by a computational network model, in which simulated responses across retinotopic arrays of orientation-selective elements were modulated by cortical association fields, represented as multiplicative kernels computed from the differences in pairwise edge statistics between target and distractor images. Comparing the experimental and the computational results suggests that each iteration of the lateral interactions takes at least ms of cortical processing time. Our results provide evidence that cortical association fields between orientation selective elements in early visual areas can account for important temporal and task-dependent aspects of the psychometric curves characterizing human contour perception, with the remaining discrepancies postulated to arise from the influence of higher cortical areas.     

Saccadic motor planning by integrating visual information and pre-information on neural dynamic fields

A functional model of target selection in the saccadic system is presented, incorporating elements of visual processing, motor planning, and motor control. We address the integration of visual information with pre-information. which is provided by manipulating the probability that a target appears at a certain location. This integration is achieved within a dynamic representation of planned eye movement which is modeled through distributions of activation on a topographic field. Visual input evokes activation, which is also constrained by lateral interaction within the field and by preshaping input representing pre-information. The model describes target selection observable in paradigms in which visual goals are presented at more than one location. Specifically, we model the transition from averaging, where endpoints of first saccades fall between two visual target locations, to decision making, where endpoints of first saccades fall accurately onto one of two simultaneously presented visual targets. We make predictions about how metrical biases of first saccades are induced by pre-information about target locations acquired by learning. When coupled to a motor control stage, activation dynamics on the planning level contribute to stabilizing gaze under fixation conditions. The neurophysiological relevance of our functional model is discussed.     

EEG Findings of Reduced Neural Synchronization during Visual Integration in Schizophrenia

Schizophrenia patients exhibit well-documented visual processing deficits. One area of disruption is visual integration, the ability to form global objects from local elements. However, most studies of visual integration in schizophrenia have been conducted in the context of an active attention task, which may influence the findings. In this study we examined visual integration using electroencephalography (EEG) in a passive task to elucidate neural mechanisms associated with poor visual integration. Forty-six schizophrenia patients and 30 healthy controls had EEG recorded while passively viewing figures comprised of real, illusory, or no contours. We examined visual P100, N100, and P200 event-related potential (ERP) components, as well as neural synchronization in the gamma (30-60 Hz) band assessed by the EEG phase locking factor (PLF). The N100 was significantly larger to illusory vs. no contour, and illusory vs. real contour stimuli while the P200 was larger only to real vs. illusory stimuli; there were no significant interactions with group. Compared to controls, patients failed to show increased phase locking to illusory versus no contours between 40-60 Hz. Also, controls, but not patients, had larger PLF between 30-40 Hz when viewing real vs. illusory contours. Finally, the positive symptom factor of the BPRS was negatively correlated with PLF values between 40-60 Hz to illusory stimuli, and with PLF between 30-40 Hz to real contour stimuli. These results suggest that the pattern of results across visual processing conditions is similar in patients and controls. However, patients have deficits in neural synchronization in the gamma range during basic processing of illusory contours when attentional demand is limited.     

Abnormal Contextual Modulation of Visual Contour Detection in Patients with Schizophrenia

Schizophrenia patients demonstrate perceptual deficits consistent with broad dysfunction in visual context processing. These include poor integration of segments forming visual contours, and reduced visual contrast effects (e.g. weaker orientation-dependent surround suppression, ODSS). Background image context can influence contour perception, as stimuli near the contour affect detection accuracy. Because of ODSS, this contextual modulation depends on the relative orientation between the contour and flanking elements, with parallel flankers impairing contour perception. However in schizophrenia, the impact of abnormal ODSS during contour perception is not clear. It is also unknown whether deficient contour perception marks genetic liability for schizophrenia, or is strictly associated with clinical expression of this disorder. We examined contour detection in 25 adults with schizophrenia, 13 unaffected first-degree biological relatives of schizophrenia patients, and 28 healthy controls. Subjects performed a psychophysics experiment designed to quantify the effect of flanker orientation during contour detection. Overall, patients with schizophrenia showed poorer contour detection performance than relatives or controls. Parallel flankers suppressed and orthogonal flankers enhanced contour detection performance for all groups, but parallel suppression was relatively weaker for schizophrenia patients than healthy controls. Relatives of patients showed equivalent performance with controls. Computational modeling suggested that abnormal contextual modulation in schizophrenia may be explained by suppression that is more broadly tuned for orientation. Abnormal flanker suppression in schizophrenia is consistent with weaker ODSS and/or broader orientation tuning. This work provides the first evidence that such perceptual abnormalities may not be associated with a genetic liability for schizophrenia.     

Perceptual organization of local elements into global shapes in the human visual cortex

The question of how local image features on the retina are integrated into perceived global shapes is central to our understanding of human visual perception. Psychophysical investigations have suggested that the emergence of a coherent visual percept, or a "good-Gestalt", is mediated by the perceptual organization of local features based on their similarity. However, the neural mechanisms that mediate unified shape perception in the human brain remain largely unknown. Using human fMRI, we demonstrate that not only higher occipitotemporal but also early retinotopic areas are involved in the perceptual organization and detection of global shapes. Specifically, these areas showed stronger fMRI responses to global contours consisting of collinear elements than to patterns of randomly oriented local elements. More importantly, decreased detection performance and fMRI activations were observed when misalignment of the contour elements disturbed the perceptual coherence of the contours. However, grouping of the misaligned contour elements by disparity resulted in increased performance and fMRI activations, suggesting that similar neural mechanisms may underlie grouping of local elements to global shapes by different visual features (orientation or disparity). Thus, these findings provide novel evidence for the role of both early feature integration processes and higher stages of visual analysis in coherent visual perception.     

See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex

Our understanding of multisensory integration has advanced because of recent functional neuroimaging studies of three areas in human lateral occipito-temporal cortex: superior temporal sulcus, area LO and area MT (V5). Superior temporal sulcus is activated strongly in response to meaningful auditory and visual stimuli, but responses to tactile stimuli have not been well studied. Area LO shows strong activation in response to both visual and tactile shape information, but not to auditory representations of objects. Area MT, an important region for processing visual motion, also shows weak activation in response to tactile motion, and a signal that drops below resting baseline in response to auditory motion. Within superior temporal sulcus, a patchy organization of regions is activated in response to auditory, visual and multisensory stimuli. This organization appears similar to that observed in polysensory areas in macaque superior temporal sulcus, suggesting that it is an anatomical substrate for multisensory integration. A patchy organization might also be a neural mechanism for integrating disparate representations within individual sensory modalities, such as representations of visual form and visual motion.     

Visual pattern discrimination as an element of the fly's orientation behaviour

Summary The visually guided orientation behaviour of stationarily flying Musca domestica (females) has been investigated. Under such conditions, the flight activity does not influence the visual stimulus (openloop) and the tendency of a fly to orientate towards some visual object can be recorded as a yaw torque reaction (orientation response).—Orientation responses to flickering stripes reveal two different mechanisms of visual integration, namely a local flicker detecting mechanism and a specific kind of dynamic lateral interactions (Figs. 3, 5). The lateral interactions are mediated by a field of interconnections of receptors which are separated by at least 4 to 6 vertical rows of ommatidia (Figs. 3, 8). While stimulation of not more than 3 vertical rows of ommatidia activates only flicker detection, stimuli of more than 6° width may in addition exert an excitatory or an inhibitory influence as a consequence of the associated nonlinear interactions (Figs. 5, 7). The relevance of these lateral interactions for tracking and chasing behaviour is discussed. It is suggested that the fly's visual pattern discrimination rests essentially on these lateral interactions.     

Dynamic shape integration in extrastriate cortex

BACKGROUND: In anorthoscopic viewing conditions, observers can perceive a moving object through a narrow slit even when only portions of its contour are visible at any time. We used fMRI to examine the contribution of early and later visual cortical areas to dynamic shape integration. Observers' success at integrating the shape of the slit-viewed object was manipulated by varying the degree to which the stimulus was dynamically distorted. Line drawings of common objects were either moderately distorted, strongly distorted, or shown undistorted. Phenomenologically, increasing the stimulus distortion made both object shape and motion more difficult to perceive.RESULTS: We found that bilateral cortical activity in portions of the ventral occipital cortex, corresponding to known object areas within the lateral occipital complex (LOC), was inversely correlated with the degree of stimulus distortion. We found that activity in left MT+, the human cortical area specialized for motion, showed a similar pattern as the ventral occipital region. The LOC also showed greater activity to a fully visible moving object than to the undistorted slit-viewed object. Area MT+, however, showed more equivalent activity to both the slit-viewed and fully visible moving objects.CONCLUSIONS: In early retinotopic cortex, the distorted and undistorted stimuli elicited the same amount of activity. Higher visual areas, however, were correlated with the percept of the coherent object, and this correlation suggests that the shape integration is mediated by later visual cortical areas. Motion information from the dorsal stream may project to the LOC to produce the shape percept.     

Integration of bimodal looming signals through neuronal coherence in the temporal lobe

The ability to integrate information across multiple sensory systems offers several behavioral advantages, from quicker reaction times and more accurate responses to better detection and more robust learning. At the neural level, multisensory integration requires large-scale interactions between different brain regions--the convergence of information from separate sensory modalities, represented by distinct neuronal populations. The interactions between these neuronal populations must be fast and flexible, so that behaviorally relevant signals belonging to the same object or event can be immediately integrated and integration of unrelated signals can be prevented. Looming signals are a particular class of signals that are behaviorally relevant for animals and that occur in both the auditory and visual domain. These signals indicate the rapid approach of objects and provide highly salient warning cues about impending impact. We show here that multisensory integration of auditory and visual looming signals may be mediated by functional interactions between auditory cortex and the superior temporal sulcus, two areas involved in integrating behaviorally relevant auditory-visual signals. Audiovisual looming signals elicited increased gamma-band coherence between these areas, relative to unimodal or receding-motion signals. This suggests that the neocortex uses fast, flexible intercortical interactions to mediate multisensory integration.     

Adult visual cortical plasticity

The visual cortex has the capacity for experience-dependent change, or cortical plasticity, that is retained throughout life. Plasticity is invoked for encoding information during perceptual learning, by internally representing the regularities of the visual environment, which is useful for facilitating intermediate-level vision-contour integration and surface segmentation. The same mechanisms have adaptive value for functional recovery after CNS damage, such as that associated with stroke or neurodegenerative disease. A common feature to plasticity in primary visual cortex (V1) is an association field that links contour elements across the visual field. The circuitry underlying the association field includes a plexus of long-range horizontal connections formed by cortical pyramidal cells. These connections undergo rapid and exuberant sprouting and pruning in response to removal of sensory input, which can account for the topographic reorganization following retinal lesions. Similar alterations in cortical circuitry may be involved in perceptual learning, and the changes observed in V1 may be representative of how learned information is encoded throughout the cerebral cortex.