Dynamic decorrelation as a unifying principle for explaining a broad range of brightness phenomena

The visual system is highly sensitive to spatial context for encoding luminance patterns. Context sensitivity inspired the proposal of many neural mechanisms for explaining the perception of luminance (brightness). Here we propose a novel computational model for estimating the brightness of many visual illusions. We hypothesize that many aspects of brightness can be explained by a dynamic filtering process that reduces the redundancy in edge representations on the one hand, while non-redundant activity is enhanced on the other. The dynamic filter is learned for each input image and implements context sensitivity. Dynamic filtering is applied to the responses of (model) complex cells in order to build a gain control map. The gain control map then acts on simple cell responses before they are used to create a brightness map via activity propagation. Our approach is successful in predicting many challenging visual illusions, including contrast effects, assimilation, and reverse contrast with the same set of model parameters.     

Computing Complex Visual Features with Retinal Spike Times

Neurons in sensory systems can represent information not only by their firing rate, but also by the precise timing of individual spikes. For example, certain retinal ganglion cells, first identified in the salamander, encode the spatial structure of a new image by their first-spike latencies. Here we explore how this temporal code can be used by downstream neural circuits for computing complex features of the image that are not available from the signals of individual ganglion cells. To this end, we feed the experimentally observed spike trains from a population of retinal ganglion cells to an integrate-and-fire model of post-synaptic integration. The synaptic weights of this integration are tuned according to the recently introduced tempotron learning rule. We find that this model neuron can perform complex visual detection tasks in a single synaptic stage that would require multiple stages for neurons operating instead on neural spike counts. Furthermore, the model computes rapidly, using only a single spike per afferent, and can signal its decision in turn by just a single spike. Extending these analyses to large ensembles of simulated retinal signals, we show that the model can detect the orientation of a visual pattern independent of its phase, an operation thought to be one of the primitives in early visual processing. We analyze how these computations work and compare the performance of this model to other schemes for reading out spike-timing information. These results demonstrate that the retina formats spatial information into temporal spike sequences in a way that favors computation in the time domain. Moreover, complex image analysis can be achieved already by a simple integrate-and-fire model neuron, emphasizing the power and plausibility of rapid neural computing with spike times.     

Colour processing in complex environments: insights from the visual system of bees

Colour vision enables animals to detect and discriminate differences in chromatic cues independent of brightness. How the bee visual system manages this task is of interest for understanding information processing in miniaturized systems, as well as the relationship between bee pollinators and flowering plants. Bees can quickly discriminate dissimilar colours, but can also slowly learn to discriminate very similar colours, raising the question as to how the visual system can support this, or whether it is simply a learning and memory operation. We discuss the detailed neuroanatomical layout of the brain, identify probable brain areas for colour processing, and suggest that there may be multiple systems in the bee brain that mediate either coarse or fine colour discrimination ability in a manner dependent upon individual experience. These multiple colour pathways have been identified along both functional and anatomical lines in the bee brain, providing us with some insights into how the brain may operate to support complex colour discrimination behaviours.     

Multi-scale lines and edges in V1 and beyond: Brightness,object categorization and recognition,and consciousness

In this paper we present an improved model for line and edge detection in cortical area V1. This model is based on responses of simple and complex cells, and it is multi-scale with no free parameters. We illustrate the use of the multi-scale line/edge representation in different processes: visual reconstruction or brightness perception, automatic scale selection and object segregation. A two-level object categorization scenario is tested in which pre-categorization is based on coarse scales only and final categorization on coarse plus fine scales. We also present a multi-scale object and face recognition model. Processing schemes are discussed in the framework of a complete cortical architecture. The fact that brightness perception and object recognition may be based on the same symbolic image representation is an indication that the entire (visual) cortex is involved in consciousness.     

A computational theory of visual receptive fields

A receptive field constitutes a region in the visual field where a visual cell or a visual operator responds to visual stimuli. This paper presents a theory for what types of receptive field profiles can be regarded as natural for an idealized vision system, given a set of structural requirements on the first stages of visual processing that reflect symmetry properties of the surrounding world. These symmetry properties include (i) covariance properties under scale changes, affine image deformations, and Galilean transformations of space–time as occur for real-world image data as well as specific requirements of (ii) temporal causality implying that the future cannot be accessed and (iii) a time-recursive updating mechanism of a limited temporal buffer of the past as is necessary for a genuine real-time system. Fundamental structural requirements are also imposed to ensure (iv) mutual consistency and a proper handling of internal representations at different spatial and temporal scales. It is shown how a set of families of idealized receptive field profiles can be derived by necessity regarding spatial, spatio-chromatic, and spatio-temporal receptive fields in terms of Gaussian kernels, Gaussian derivatives, or closely related operators. Such image filters have been successfully used as a basis for expressing a large number of visual operations in computer vision, regarding feature detection, feature classification, motion estimation, object recognition, spatio-temporal recognition, and shape estimation. Hence, the associated so-called scale-space theory constitutes a both theoretically well-founded and general framework for expressing visual operations. There are very close similarities between receptive field profiles predicted from this scale-space theory and receptive field profiles found by cell recordings in biological vision. Among the family of receptive field profiles derived by necessity from the assumptions, idealized models with very good qualitative agreement are obtained for (i) spatial on-center/off-surround and off-center/on-surround receptive fields in the fovea and the LGN, (ii) simple cells with spatial directional preference in V1, (iii) spatio-chromatic double-opponent neurons in V1, (iv) space–time separable spatio-temporal receptive fields in the LGN and V1, and (v) non-separable space–time tilted receptive fields in V1, all within the same unified theory. In addition, the paper presents a more general framework for relating and interpreting these receptive fields conceptually and possibly predicting new receptive field profiles as well as for pre-wiring covariance under scaling, affine, and Galilean transformations into the representations of visual stimuli. This paper describes the basic structure of the necessity results concerning receptive field profiles regarding the mathematical foundation of the theory and outlines how the proposed theory could be used in further studies and modelling of biological vision. It is also shown how receptive field responses can be interpreted physically, as the superposition of relative variations of surface structure and illumination variations, given a logarithmic brightness scale, and how receptive field measurements will be invariant under multiplicative illumination variations and exposure control mechanisms.     

Temporal Statistics of Natural Image Sequences Generated by Movements with Insect Flight Characteristics

Many flying insects, such as flies, wasps and bees, pursue a saccadic flight and gaze strategy. This behavioral strategy is thought to separate the translational and rotational components of self-motion and, thereby, to reduce the computational efforts to extract information about the environment from the retinal image flow. Because of the distinguishing dynamic features of this active flight and gaze strategy of insects, the present study analyzes systematically the spatiotemporal statistics of image sequences generated during saccades and intersaccadic intervals in cluttered natural environments. We show that, in general, rotational movements with saccade-like dynamics elicit fluctuations and overall changes in brightness, contrast and spatial frequency of up to two orders of magnitude larger than translational movements at velocities that are characteristic of insects. Distinct changes in image parameters during translations are only caused by nearby objects. Image analysis based on larger patches in the visual field reveals smaller fluctuations in brightness and spatial frequency composition compared to small patches. The temporal structure and extent of these changes in image parameters define the temporal constraints imposed on signal processing performed by the insect visual system under behavioral conditions in natural environments.     

Motion processing with wide-field neurons in the retino-tecto-rotundal pathway

The retino-tecto-rotundal pathway is the main visual pathway in non-mammalian vertebrates and has been found to be highly involved in visual processing. Despite the extensive receptive fields of tectal and rotundal wide-field neurons, pattern discrimination tasks suggest a system with high spatial resolution. In this paper, we address the problem of how global processing performed by motion-sensitive wide-field neurons can be brought into agreement with the concept of a local analysis of visual stimuli. As a solution to this problem, we propose a firing-rate model of the retino-tecto-rotundal pathway which describes how spatiotemporal information can be organized and retained by tectal and rotundal wide-field neurons while processing Fourier-based motion in absence of periodic receptive-field structures. The model incorporates anatomical and electrophysiological experimental data on tectal and rotundal neurons, and the basic response characteristics of tectal and rotundal neurons to moving stimuli are captured by the model cells. We show that local velocity estimates may be derived from rotundal-cell responses via superposition in a subsequent processing step. Experimentally testable predictions which are both specific and characteristic to the model are provided. Thus, a conclusive explanation can be given of how the retino-tecto-rotundal pathway enables the animal to detect and localize moving objects or to estimate its self-motion parameters.     

Representation of visual stimuli in inferior temporal cortex.

In primates, inferior temporal (IT) cortex is crucial for the processing and storage of visual information about form and colour. This article reviews the properties of IT neurons and considers how these properties may underlie the perceptual and mnemonic functions of IT cortex. The available evidence suggests that the processing of the facial image by IT cortex is similar to its processing of other visual patterns. Faces and other complex visual stimuli appear to be represented by the pattern of responses over a population of IT neurons rather than by the responses of specific 'feature detectors' or 'grandmother' cells. IT neurons with adult-like stimulus properties are present in monkeys as young as six weeks old.     

Orientation enhancement in early visual processing can explain time course of brightness contrast and White’s illusion

Dynamics of orientation tuning in V1 indicates that computational model of V1 should not only comprise of bank of static spatially oriented filters but also include the contribution for dynamical response facilitation or suppression along orientation. Time evolution of orientation response in V1 can emerge due to time- dependent excitation and lateral inhibition in the orientation domain. Lateral inhibition in the orientation domain suggests that Ernst Mach’s proposition can be applied for the enhancement of initial orientation distribution that is generated due to interaction of visual stimulus with spatially oriented filters and subcortical temporal filter. Oriented spatial filtering that appears much early ( $<$ 70 ms) in the sequence of visual information processing can account for many of the brightness illusions observed at steady state. It is therefore expected that time evolution of orientation response might be reflecting in the brightness percept over time. Our numerical study suggests that only spatio-temporal filtering at early phase can explain experimentally observed temporal dynamics of brightness contrast illusion. But, enhancement of orientation response at early phase of visual processing is the key mechanism that can guide visual system to predict the brightness by “Max-rule” or “Winner Takes All” (WTA) estimation and thus producing White’s illusions at any exposure.     

Melanopsin-based brightness discrimination in mice and humans

Photoreception in the mammalian retina is not restricted to rods and cones but extends to a small number of intrinsically photoreceptive retinal ganglion cells (ipRGCs), expressing the photopigment melanopsin. ipRGCs are known to support various accessory visual functions including circadian photoentrainment and pupillary reflexes. However, despite anatomical and physiological evidence that they contribute to the thalamocortical visual projection, no aspect of visual discrimination has been shown to rely upon ipRGCs. Based on their currently known roles, we hypothesized that ipRGCs may contribute to distinguishing brightness. This percept is related to an object's luminance-a photometric measure of light intensity relevant for cone photoreceptors. However, the perceived brightness of different sources is not always predicted by their respective luminance. Here, we used parallel behavioral and electrophysiological experiments to first show that melanopsin contributes to brightness discrimination in both retinally degenerate and fully sighted mice. We continued to use comparable paradigms in psychophysical experiments to provide evidence for a similar role in healthy human subjects. These data represent the first direct evidence that an aspect of visual discrimination in normally sighted subjects can be supported by inner retinal photoreceptors.     

Human wavelength discrimination of monochromatic light explained by optimal wavelength decoding of light of unknown intensity

We show that human ability to discriminate the wavelength of monochromatic light can be understood as maximum likelihood decoding of the cone absorptions, with a signal processing efficiency that is independent of the wavelength. This work is built on the framework of ideal observer analysis of visual discrimination used in many previous works. A distinctive aspect of our work is that we highlight a perceptual confound that observers should confuse a change in input light wavelength with a change in input intensity. Hence a simple ideal observer model which assumes that an observer has a full knowledge of input intensity should over-estimate human ability in discriminating wavelengths of two inputs of unequal intensity. This confound also makes it difficult to consistently measure human ability in wavelength discrimination by asking observers to distinguish two input colors while matching their brightness. We argue that the best experimental method for reliable measurement of discrimination thresholds is the one of Pokorny and Smith, in which observers only need to distinguish two inputs, regardless of whether they differ in hue or brightness. We mathematically formulate wavelength discrimination under this wavelength-intensity confound and show a good agreement between our theoretical prediction and the behavioral data. Our analysis explains why the discrimination threshold varies with the input wavelength, and shows how sensitively the threshold depends on the relative densities of the three types of cones in the retina (and in particular predict discriminations in dichromats). Our mathematical formulation and solution can be applied to general problems of sensory discrimination when there is a perceptual confound from other sensory feature dimensions.     

Colour and brightness coding in the central nervous system: theoretical aspects and visual evoked potentials to homogeneous red and green stimuli

We designed visual evoked potentials experiments to study the differential aspects of colour and brightness coding in man. The substitution of equally bright red and green stimuli for a background yellow was investigated and compared with different luminance increments and decrements of red and green. A dominant N87 component was found for a colour change from yellow to brighter red colours, which was less pronounced for green and absent for yellow luminance changes. It is also absent for pure red luminance increments and green luminance changes, but reappears with red luminance decrements or red-offset. The data are discussed within the framework of a new concept of how the visual system fuses red-green information and black-white border information. Retinal X-cells can transmit colour and high spatial frequency achromatic information simultaneously by encoding only the presence of edges (a.c.) for the black-white stimuli and the presence of both edges (a.c.) and uniform areas of colour (d.c.) for red-green stimuli. Phylogenetically this kind of information transmission enables colour vision to be implemented in a retina such as the cat's by adding only a second class of cones. Barlow's economy principle will be violated for colour in the periphery, but restored early in the striate cortex where there is an early decoding of the combined chromatic and achromatic information by the concentric double opponent cells. The N87 behaviour correlates with the proposed discharge of peripheral X-type cells, but not with the discharge of cortical double opponent concentric or simple cells, which no longer respond to homogeneous colour stimuli. It is suggested that N87 may be generated by geniculate afferents in the dendritic arborization of cortical cells, reflecting the behaviour of peripheral units, and thus the violation of the economy principle, rather than the next step in cortical processing. The early cortical restoration of the economy principle is supported by the absence of any further dissociated behaviour for colour and brightness in later components.     

Scanning image correlation spectroscopy

Molecular interactions are at the origin of life. How molecules get at different locations in the cell and how they locate their partners is a major and partially unresolved question in biology that is paramount to signaling. Spatio-temporal correlations of fluctuating fluorescently tagged molecules reveal how they move, interact, and bind in the different cellular compartments. Methods based on fluctuations represent a remarkable technical advancement in biological imaging. Here we discuss image analysis methods based on spatial and temporal correlation of fluctuations, raster image correlation spectroscopy, number and brightness, and spatial cross-correlations that give us information about how individual molecules move in cells and interact with partners at the single molecule level. These methods can be implemented with a standard laser scanning microscope and produce a cellular level spatio-temporal map of molecular interactions.     

Temporal codes and sparse representations: A key to understanding rapid processing in the visual system

Where neural information processing is concerned, there is no debate about the fact that spikes are the basic currency for transmitting information between neurons. How the brain actually uses them to encode information remains more controversial. It is commonly assumed that neuronal firing rate is the key variable, but the speed with which images can be analysed by the visual system poses a major challenge for rate-based approaches. We will thus expose here the possibility that the brain makes use of the spatio-temporal structure of spike patterns to encode information. We then consider how such rapid selective neural responses can be generated rapidly through spike-timing-dependent plasticity (STDP) and how these selectivities can be used for visual representation and recognition. Finally, we show how temporal codes and sparse representations may very well arise one from another and explain some of the remarkable features of processing in the visual system.     

Visually guided orientation in flies: case studies in computational neuroethology

To understand the functioning of nervous systems and, in particular, how they control behaviour we must bridge many levels of complexity from molecules, cells and synapses to perception behaviour. Although experimental analysis is a precondition for understanding by nervous systems, it is in no way sufficient. The understanding is aided at all levels of complexity by modelling. Modelling proved to be an inevitable tool to test the experimentally established hypotheses. In this review it will by exemplified by three case studies that the appropriate level of modelling needs to be adjusted to the particular computational problems that are to be solved. (1) Specific features of the highly virtuosic pursuit behaviour of male flies can be understood on the basis of a phenomenological model that relates the visual input to the motor output. (2) The processing of retinal image motion as is experienced by freely moving animals can be understood on the basis of a model consisting of algorithmic components and components which represent a simple equivalent circuit of nerve cells. (3) Behaviourally relevant features of the reliability of encoding of visual motion information can be understood by modelling the transformation of postsynaptic potentials into sequences of spike trains.     

Molecular modelling methods for prediction of sequence-selectivity in DNA recognition

We describe how one can apply molecular modelling methods, based on the molecular mechanics/generalised Born (MM/GB) approach, to the prediction of the relative affinity of DNA minor groove binding ligands for different DNA sequences. We discuss the theoretical background to the technique, some variations in the methodology that can be employed, and illustrate its application through a case study: analysis of the energetics of binding of Hoechst 33258 to the minor groove of various A/T-rich DNA duplexes. We show how the underpinning molecular dynamics (MD) simulations can be set up, how they can be analysed for satisfactory behaviour, and various approaches to extracting thermodynamics of drug binding from them. We find that while certain elaborations to the basic MM/GB method can improve the agreement with experimental data (e.g., calculating the DNA perturbation energy), others have to be analysed with more caution (e.g., calculating configurational entropy changes). Overall, these methodologies can rank the affinity of a ligand for the minor groove of different DNA sequences fairly well, but the calculation of absolute binding affinities is not very reliable.     

On the role of medial geometry in human vision.

A key challenge underlying theories of vision is how the spatially restricted, retinotopically represented feature analysis can be integrated to form abstract, coordinate-free object models. A resolution likely depends on the use of intermediate-level representations which can on the one hand be populated by local features and on the other hand be used as atomic units underlying the formation of, and interaction with, object hypotheses. The precise structure of this intermediate representation derives from the varied requirements of a range of visual tasks which motivate a significant role for incorporating a geometry of visual form. The need to integrate input from features capturing surface properties such as texture, shading, motion, color, etc., as well as from features capturing surface discontinuities such as silhouettes, T-junctions, etc., implies a geometry which captures both regional and boundary aspects. Curves, as a geometric model of boundaries, have been extensively used as an intermediate representation in computational, perceptual, and physiological studies, while the use of the medial axis (MA) has been popular mainly in computer vision as a geometric region-based model of the interior of closed boundaries. We extend the traditional model of the MA to represent images, where each MA segment represents a region of the image which we call a visual fragment. We present a unified theory of perceptual grouping and object recognition where through various sequences of transformations of the MA representation, visual fragments are grouped in various configurations to form object hypotheses, and are related to stored models. The mechanisms underlying both the computation and the transformation of the MA is a lateral wave propagation model. Recent psychophysical experiments depicting contrast sensitivity map peaks at the medial axes of stimuli, and experiments on perceptual filling-in, and brightness induction and modulation, are consistent with both the use of an MA representation and a propagation-based scheme. Also, recent neurophysiological recordings in V1 correlate with the MA hypothesis and a horizontal propagation scheme. This evidence supports a geometric computational paradigm for processing sensory data where both dynamic in-plane propagation and feedforward-feedback connections play an integral role.     

Computational modeling and exploration of contour integration for visual saliency

We propose a computational model of contour integration for visual saliency. The model uses biologically plausible devices to simulate how the representations of elements aligned collinearly along a contour in an image are enhanced. Our model adds such devices as a dopamine-like fast plasticity, local GABAergic inhibition and multi-scale processing of images. The fast plasticity addresses the problem of how neurons in visual cortex seem to be able to influence neurons they are not directly connected to, for instance, as observed in contour closure effect. Local GABAergic inhibition is used to control gain in the system without using global mechanisms which may be non-plausible given the limited reach of axonal arbors in visual cortex. The model is then used to explore not only its validity in real and artificial images, but to discover some of the mechanisms involved in processing of complex visual features such as junctions and end-stops as well as contours. We present evidence for the validity of our model in several phases, starting with local enhancement of only a few collinear elements. We then test our model on more complex contour integration images with a large number of Gabor elements. Sections of the model are also extracted and used to discover how the model might relate contour integration neurons to neurons that process end-stops and junctions. Finally, we present results from real world images. Results from the model suggest that it is a good current approximation of contour integration in human vision. As well, it suggests that contour integration mechanisms may be strongly related to mechanisms for detecting end-stops and junction points. Additionally, a contour integration mechanism may be involved in finding features for objects such as faces. This suggests that visual cortex may be more information efficient and that neural regions may have multiple roles.     

Fluorogen-activating single-chain antibodies for imaging cell surface proteins

Imaging of live cells has been revolutionized by genetically encoded fluorescent probes, most famously green and other fluorescent proteins, but also peptide tags that bind exogenous fluorophores. We report here the development of protein reporters that generate fluorescence from otherwise dark molecules (fluorogens). Eight unique fluorogen activating proteins (FAPs) have been isolated by screening a library of human single-chain antibodies (scFvs) using derivatives of thiazole orange and malachite green. When displayed on yeast or mammalian cell surfaces, these FAPs bind fluorogens with nanomolar affinity, increasing green or red fluorescence thousands-fold to brightness levels typical of fluorescent proteins. Spectral variation can be generated by combining different FAPs and fluorogen derivatives. Visualization of FAPs on the cell surface or within the secretory apparatus of mammalian cells can be achieved by choosing membrane permeant or impermeant fluorogens. The FAP technique is extensible to a wide variety of nonfluorescent dyes.     

Mapping landscape corridors

Corridors are important geographic features for biological conservation and biodiversity assessment. The identification and mapping of corridors is usually based on visual interpretations of movement patterns (functional corridors) or habitat maps (structural corridors). We present a method for automated corridor mapping with morphological image processing, and demonstrate the approach with a forest map derived from satellite imagery of northern Slovakia. We show how the approach can be used to differentiate between relatively narrow (‘line’) and wide (‘strip’) structural corridors by mapping corridors at multiple scales of observation, and indicate how to map functional corridors with maps of observed or simulated organism movement. An application to environmental reporting is demonstrated by assessing structural forest corridors in relation to forest types in northern Slovakia.