A laminar cortical model of stereopsis and 3D surface perception: closure and da Vinci stereopsis

A laminar cortical model of stereopsis and 3D surface perception is developed and simulated. The model describes how monocular and binocular oriented filtering interact with later stages of 3D boundary formation and surface filling-in in the LGN and cortical areas V1, V2, and V4. It proposes how interactions between layers 4, 3B, and 2/3 in V1 and V2 contribute to stereopsis, and how binocular and monocular information combine to form 3D boundary and surface representations. The model includes two main new developments: (1) It clarifies how surface-to-boundary feedback from V2 thin stripes to pale stripes helps to explain data about stereopsis. This feedback has previously been used to explain data about 3D figure-ground perception. (2) It proposes that the binocular false match problem is subsumed under the Gestalt grouping problem. In particular, the disparity filter, which helps to solve the correspondence problem by eliminating false matches, is realized using inhibitory interneurons as part of the perceptual grouping process by horizontal connections in layer 2/3 of cortical area V2. The enhanced model explains all the psychophysical data previously simulated by Grossberg and Howe (2003), such as contrast variations of dichoptic masking and the correspondence problem, the effect of interocular contrast differences on stereoacuity, Panum's limiting case, the Venetian blind illusion, stereopsis with polarity-reversed stereograms, and da Vinci stereopsis. It also explains psychophysical data about perceptual closure and variations of da Vinci stereopsis that previous models cannot yet explain.     

Natural image statistics mediate brightness 'filling in'

Although the human visual system can accurately estimate the reflectance (or lightness) of surfaces under enormous variations in illumination, two equiluminant grey regions can be induced to appear quite different simply by placing a light-dark luminance transition between them. This illusion, the Craik-Cornsweet-O'Brien (CCOB) effect, has been taken as evidence for a low-level 'filling-in' mechanism subserving lightness perception. Here, we present evidence that the mechanism responsible for the CCOB effect operates not via propagation of a neural signal across space but by amplification of the low spatial frequency (SF) structure of the image. We develop a simple computational model that relies on the statistics of natural scenes actively to reconstruct the image that is most likely to have caused an observed series of responses across SF channels. This principle is tested psychophysically by deriving classification images (CIs) for subjects' discrimination of the contrast polarity of CCOB stimuli masked with noise. CIs resemble 'filled-in' stimuli; i.e. observers rely on portions of the stimuli that contain no information per se but that correspond closely to the reported perceptual completion. As predicted by the model, the filling-in process is contingent on the presence of appropriate low SF structure.     

The dynamic range of human lightness perception

Natural viewing challenges the visual system with images that have a dynamic range of light intensity (luminance) that can approach 1,000,000:1 and that often exceeds 10,000:1 [1, 2]. The range of perceived surface reflectance (lightness), however, can be well approximated by the Munsell matte neutral scale (N 2.0/ to N 9.5/), consisting of surfaces whose reflectance varies by about 30:1. Thus, the visual system must map a large range of surface luminance onto a much smaller range of surface lightness. We measured this mapping in images with a dynamic range close to that of natural images. We studied simple images that lacked segmentation cues that would indicate multiple regions of illumination. We found a remarkable degree of compression: at a single image location, a stimulus luminance range of 5,905:1 can be mapped onto an extended lightness scale that has a reflectance range of 100:1. We characterized how the luminance-to-lightness mapping changes with stimulus context. Our data rule out theories that predict perceived lightness from luminance ratios or Weber contrast. A mechanistic model connects our data to theories of adaptation and provides insight about how the underlying visual response varies with context.     

Macaque Monkeys Perceive the Flash Lag Illusion

Transmission of neural signals in the brain takes time due to the slow biological mechanisms that mediate it. During such delays, the position of moving objects can change substantially. The brain could use statistical regularities in the natural world to compensate neural delays and represent moving stimuli closer to real time. This possibility has been explored in the context of the flash lag illusion, where a briefly flashed stimulus in alignment with a moving one appears to lag behind the moving stimulus. Despite numerous psychophysical studies, the neural mechanisms underlying the flash lag illusion remain poorly understood, partly because it has never been studied electrophysiologically in behaving animals. Macaques are a prime model for such studies, but it is unknown if they perceive the illusion. By training monkeys to report their percepts unbiased by reward, we show that they indeed perceive the illusion qualitatively similar to humans. Importantly, the magnitude of the illusion is smaller in monkeys than in humans, but it increases linearly with the speed of the moving stimulus in both species. These results provide further evidence for the similarity of sensory information processing in macaques and humans and pave the way for detailed neurophysiological investigations of the flash lag illusion in behaving macaques.     

Adelson's tile and snake illusions: a Helmholtzian type of simultaneous lightness contrast

Adelson's tile, snake, and some other lightness illusions of the same type were measured with the Munsell neutral scale for twenty observers. It was shown that theories based on low-level luminance contrast processing could hardly explain these illusions. Neither can those based on luminance X-junctions. On the other hand, Helmholtz's idea, that simultaneous lightness contrast originates from an error in judgement of apparent illumination, has been elaborated so as to account for the tile and snake illusions as well as other demonstrations presented in this report.     

Quantification of phase synchronization phenomena and their importance for verbal memory processes

The tangential neurons in the lobula plate region of the flies are known to respond to visual motion across broad receptive fields in visual space.When intracellular recordings are made from tangential neurons while the intact animal is stimulated visually with moving natural imagery,we find that neural response depends upon speed of motion but is nearly invariant with respect to variations in natural scenery. We refer to this invariance as velocity constancy. It is remarkable because natural scenes, in spite of similarities in spatial structure, vary considerably in contrast, and contrast dependence is a feature of neurons in the early visual pathway as well as of most models for the elementary operations of visual motion detection. Thus, we expect that operations must be present in the processing pathway that reduce contrast dependence in order to approximate velocity constancy.We consider models for such operations, including spatial filtering, motion adaptation, saturating nonlinearities, and nonlinear spatial integration by the tangential neurons themselves, and evaluate their effects in simulations of a tangential neuron and precursor processing in response to animated natural imagery. We conclude that all such features reduce interscene variance in response, but that the model system does not approach velocity constancy as closely as the biological tangential cell.     

A Unified Account of Perceptual Layering and Surface Appearance in Terms of Gamut Relativity

When we look at the world—or a graphical depiction of the world—we perceive surface materials (e.g. a ceramic black and white checkerboard) independently of variations in illumination (e.g. shading or shadow) and atmospheric media (e.g. clouds or smoke). Such percepts are partly based on the way physical surfaces and media reflect and transmit light and partly on the way the human visual system processes the complex patterns of light reaching the eye. One way to understand how these percepts arise is to assume that the visual system parses patterns of light into layered perceptual representations of surfaces, illumination and atmospheric media, one seen through another. Despite a great deal of previous experimental and modelling work on layered representation, however, a unified computational model of key perceptual demonstrations is still lacking. Here we present the first general computational model of perceptual layering and surface appearance—based on a boarder theoretical framework called gamut relativity—that is consistent with these demonstrations. The model (a) qualitatively explains striking effects of perceptual transparency, figure-ground separation and lightness, (b) quantitatively accounts for the role of stimulus- and task-driven constraints on perceptual matching performance, and (c) unifies two prominent theoretical frameworks for understanding surface appearance. The model thereby provides novel insights into the remarkable capacity of the human visual system to represent and identify surface materials, illumination and atmospheric media, which can be exploited in computer graphics applications.     

Lightness, illumination, and gradients

The illumination interpretation approach claims that lightness illusions can be explained as misapplications of lightness constancy mechanisms, processes which usually enable veridical extraction of surface reflectance from luminance distributions by discounting illumination. In particular, luminance gradients are thought to provide cues about the interactions of light and surfaces. Several examples of strong lightness illusions are discussed for which explanations based on illumination interpretation can be proposed. In criticisms of this approach, a variety of demonstrations of similarly structured control displays are presented, which involve equivalent lightness effects that cannot readily be accounted for by illumination interpretation mechanisms. Furthermore, a number of known and novel displays are presented that demonstrate effects of gradients on the qualitative appearance of uniform regions. Finally, some simple simulations of neural effects of luminance distributions are discussed.     

Color and lightness constancy in different perceptual tasks

Color and lightness constancy with respect to changing illumination was studied with three different perceptual tasks: ranking of colored papers according (1) to their lightness and (2) to their chromatic similarity in photopic, mesopic, and scotopic states of adaptation, and (3) recognition of remembered colored papers after changes of illumination in photopic vision. Constancy was found in the second task, only. Excitations of light receptors and luminance channels were computed to simulate the empirical rank orders. Results of the first task can be predicted with the hypothesis that luminance channels are activated, if lightness is asked for. Sequences arranged with respect to chromatic similarity were found independent of the illuminant spectra, even if the calculated rank orders of cone excitation were changed in the altered illumination. Received: 4 October 1997 / Accepted in revised form: 26 August 1998     

The cutaneous rabbit illusion affects human primary sensory cortex somatotopically

We used functional magnetic resonance imaging (fMRI) to study neural correlates of a robust somatosensory illusion that can dissociate tactile perception from physical stimulation. Repeated rapid stimulation at the wrist, then near the elbow, can create the illusion of touches at intervening locations along the arm, as if a rabbit hopped along it. We examined brain activity in humans using fMRI, with improved spatial resolution, during this version of the classic cutaneous rabbit illusion. As compared with control stimulation at the same skin sites (but in a different order that did not induce the illusion), illusory sequences activated contralateral primary somatosensory cortex, at a somatotopic location corresponding to the filled-in illusory perception on the forearm. Moreover, the amplitude of this somatosensory activation was comparable to that for veridical stimulation including the intervening position on the arm. The illusion additionally activated areas of premotor and prefrontal cortex. These results provide direct evidence that illusory somatosensory percepts can affect primary somatosensory cortex in a manner that corresponds somatotopically to the illusory percept.     

The art of seeing and painting

The human urge to represent the three-dimensional world using two-dimensional pictorial representations dates back at least to Paleolithic times. Artists from ancient to modern times have struggled to understand how a few contours or color patches on a flat surface can induce mental representations of a three-dimensional scene. This article summarizes some of the recent breakthroughs in scientifically understanding how the brain sees that shed light on these struggles. These breakthroughs illustrate how various artists have intuitively understood paradoxical properties about how the brain sees, and have used that understanding to create great art. These paradoxical properties arise from how the brain forms the units of conscious visual perception; namely, representations of three-dimensional boundaries and surfaces. Boundaries and surfaces are computed in parallel cortical processing streams that obey computationally complementary properties. These streams interact at multiple levels to overcome their complementary weaknesses and to transform their complementary properties into consistent percepts. The article describes how properties of complementary consistency have guided the creation of many great works of art.     

Running as fast as it can: How spiking dynamics form object groupings in the laminar circuits of visual cortex

How spiking neurons cooperate to control behavioral processes is a fundamental problem in computational neuroscience. Such cooperative dynamics are required during visual perception when spatially distributed image fragments are grouped into emergent boundary contours. Perceptual grouping is a challenge for spiking cells because its properties of collinear facilitation and analog sensitivity occur in response to binary spikes with irregular timing across many interacting cells. Some models have demonstrated spiking dynamics in recurrent laminar neocortical circuits, but not how perceptual grouping occurs. Other models have analyzed the fast speed of certain percepts in terms of a single feedforward sweep of activity, but cannot explain other percepts, such as illusory contours, wherein perceptual ambiguity can take hundreds of milliseconds to resolve by integrating multiple spikes over time. The current model reconciles fast feedforward with slower feedback processing, and binary spikes with analog network-level properties, in a laminar cortical network of spiking cells whose emergent properties quantitatively simulate parametric data from neurophysiological experiments, including the formation of illusory contours; the structure of non-classical visual receptive fields; and self-synchronizing gamma oscillations. These laminar dynamics shed new light on how the brain resolves local informational ambiguities through the use of properly designed nonlinear feedback spiking networks which run as fast as they can, given the amount of uncertainty in the data that they process.     

Responses to lightness variations in early human visual cortex

Lightness is the apparent reflectance of a surface, and it depends not only on the actual luminance of the surface but also on the context in which the surface is viewed [1-10]. The cortical mechanisms of lightness processing are largely unknown, and the role of early cortical areas is still a matter of debate [11-17]. We studied the cortical responses to lightness variations in early stages of the human visual system with functional magnetic resonance imaging (fMRI) while observers were performing a demanding fixation task. The set of dynamically presented visual stimuli included the rectangular version of the classic Craik-O'Brien stimulus [3, 18, 19] and a variant that led to a weaker lightness effect, as well as a pattern with actual luminance variations. We found that the cortical activity in retinotopic areas, including the primary visual cortex (V1), is correlated with context-dependent lightness variations.     

Distinct mechanisms mediate visual detection and identification

A core organizing principle for studies of the brain is that distinct neural pathways mediate distinct behavioral tasks [1, 2]. When two related tasks are mediated by a common pathway, studies of one are likely to generalize to the other. Here, we test whether performance on two laboratory tasks that model object detection and identification are mediated by common mechanisms of visual adaptation. Although both tasks rely on the luminance pattern in images, their demands on visual processing are quite different. Object detection requires discriminating image luminance differences associated with the light reflected from adjacent objects. To encode these differences reliably, neurons adapt their limited dynamic range to prevailing viewing conditions [3-6]. Object identification, on the other hand, demands a fixed response to light reflected from an object independent of illumination [7]. We compared performance in discrimination and identification tasks for simulated surfaces. In striking contrast to studies with less structured contexts, we found clear evidence that distinct processes mediate judgments in the two tasks. These results challenge models that account for perceived lightness entirely through the action of image-encoding mechanisms.     

Effect of pictorial depth cues, binocular disparity cues and motion parallax depth cues on lightness perception in three-dimensional virtual scenes

Background

Surface lightness perception is affected by scene interpretation. There is some experimental evidence that perceived lightness under bi-ocular viewing conditions is different from perceived lightness in actual scenes but there are also reports that viewing conditions have little or no effect on perceived color. We investigated how mixes of depth cues affect perception of lightness in three-dimensional rendered scenes containing strong gradients of illumination in depth.

Methodology/Principal Findings

Observers viewed a virtual room (4 m width×5 m height×17.5 m depth) with checkerboard walls and floor. In four conditions, the room was presented with or without binocular disparity (BD) depth cues and with or without motion parallax (MP) depth cues. In all conditions, observers were asked to adjust the luminance of a comparison surface to match the lightness of test surfaces placed at seven different depths (8.5–17.5 m) in the scene. We estimated lightness versus depth profiles in all four depth cue conditions. Even when observers had only pictorial depth cues (no MP, no BD), they partially but significantly discounted the illumination gradient in judging lightness. Adding either MP or BD led to significantly greater discounting and both cues together produced the greatest discounting. The effects of MP and BD were approximately additive. BD had greater influence at near distances than far.

Conclusions/Significance

These results suggest the surface lightness perception is modulated by three-dimensional perception/interpretation using pictorial, binocular-disparity, and motion-parallax cues additively. We propose a two-stage (2D and 3D) processing model for lightness perception.     

Vague-to-crisp dynamics of percept formation modeled as operant (selectionist) process

We model the vague-to-crisp dynamics of forming percepts in the brain by combining two methodologies: dynamic logic (DL) and operant learning process. Forming percepts upon the presentation of visual inputs is likened to model selection based on sampled evidence. Our framework utilizes the DL in selecting the correct “percept” among competing ones, but uses an intrinsic reward mechanism to allow stochastic online update in lieu of performing the optimization step of the DL framework. We discuss the connection of our framework with cognitive processing and the intentional neurodynamic cycle.     

Visible Persistence of Single-Transient Random Dot Patterns: Spatial Parameters Affect the Duration of Fading Percepts

Visible persistence refers to the continuation of visual perception after the physical termination of a stimulus. We studied an extreme case of visible persistence by presenting two matrices of randomly distributed black and white pixels in succession. On the transition from one matrix to the second, the luminance polarity of all pixels within a disk- or annulus-shaped area reversed, physically creating a single second-order transient signal. This transient signal produces the percept of a disk or an annulus with an abrupt onset and a gradual offset. To study the nature of this fading percept we varied spatial parameters, such as the inner and the outer diameter of annuli (Experiment I) and the radius and eccentricity of disks (Experiment III), and measured the duration of visible persistence by having subjects adjust the synchrony of the onset of a reference stimulus with the onset or the offset of the fading percept. We validated this method by comparing two modalities of the reference stimuli (Experiment I) and by comparing the judgments of fading percepts with the judgments of stimuli that actually fade in luminance contrast (Experiment II). The results show that (i) irrespective of the reference modality, participants are able to precisely judge the on- and the offsets of the fading percepts, (ii) auditory reference stimuli lead to higher visible persistence durations than visual ones, (iii) visible persistence duration increases with the thickness of annuli and the diameter of disks, but decreases with the diameter of annuli, irrespective of stimulus eccentricity. These effects cannot be explained by stimulus energy, which suggests that more complex processing mechanisms are involved. Seemingly contradictory effects of disk and annulus diameter can be unified by assuming an abstract filling-in mechanism that speeds up with the strength of the edge signal and takes more time the larger the stimulus area is.     

Is neural filling-in necessary to explain the perceptual completion of motion and depth information?

Retinal activity is the first stage of visual perception. Retinal sampling is non-uniform and not continuous, yet visual experience is not characterized by holes and discontinuities in the world. How does the brain achieve this perceptual completion? Fifty years ago, it was suggested that visual perception involves a two-stage process of (i) edge detection followed by (ii) neural filling-in of surface properties. We examine whether this general hypothesis can account for the specific example of perceptual completion of a small target surrounded by dynamic dots (an ''artificial scotoma''), a phenomenon argued to provide insight into the mechanisms responsible for perception. We degrade the target''s borders using first blur and then depth continuity, and find that border degradation does not influence time to target disappearance. This indicates that important information for the continuity of target perception is conveyed at a coarse spatial scale. We suggest that target disappearance could result from adaptation that is not specific to borders, and question the need to hypothesize an active filling-in process to explain this phenomenon.     

Visual filling-in for computing perceptual surface properties

 The visual system is constantly confronted with the problem of integrating local signals into more global arrangements. This arises from the nature of early cell responses, whether they signal localized measures of luminance, motion, retinal position differences, or discontinuities. Consequently, from sparse, local measurements, the visual system must somehow generate the most likely hypothesis that is consistent with them. In this paper, we study the problem of determining achromatic surface properties, namely brightness. Mechanisms of brightness filling-in have been described by qualitative as well as quantitative models, such as by the one proposed by Cohen and Grossberg [Cohen and Grossberg (1984) Percept Psychophys 36: 428–456]. We demonstrate that filling-in from contrast estimates leads to a regularized solution for the computational problem of generating brightness representations from sparse estimates. This provides deeper insights into the nature of filling-in processes and the underlying objective function one wishes to compute. This particularly guided the proposal of a new modified version of filling-in, namely confidence-based filling-in which generates more robust brightness representations. Our investigation relates the modeling of perceptual data for biological vision to the mathematical frameworks of regularization theory and linear spatially variant diffusion. It therefore unifies different research directions that have so far coexisted in different scientific communities. Received: 11 March 1999 / Accepted in revised form: 14 March 2001     

The neural mechanisms of perceptual filling-in

Filling-in is a perceptual phenomenon in which a visual attribute such as colour, brightness, texture or motion is perceived in a region of the visual field even though such an attribute exists only in the surround. Filling-in dramatically reveals the dissociation between the retinal input and the percept, and raises fundamental questions about how these two relate to each other. Filling-in is observed in various situations, and is an essential part of our normal surface perception. Here, I review recent experiments examining brain activities associated with filling-in, and discuss possible neural mechanisms underlying this remarkable perceptual phenomenon. The evidence shows that neuronal activities in early visual cortical areas are involved in filling-in, providing new insights into visual cortical functions.