A comprehensive model of audiovisual perception: both percept and temporal dynamics

The sparse information captured by the sensory systems is used by the brain to apprehend the environment, for example, to spatially locate the source of audiovisual stimuli. This is an ill-posed inverse problem whose inherent uncertainty can be solved by jointly processing the information, as well as introducing constraints during this process, on the way this multisensory information is handled. This process and its result--the percept--depend on the contextual conditions perception takes place in. To date, perception has been investigated and modeled on the basis of either one of two of its dimensions: the percept or the temporal dynamics of the process. Here, we extend our previously proposed audiovisual perception model to predict both these dimensions to capture the phenomenon as a whole. Starting from a behavioral analysis, we use a data-driven approach to elicit a bayesian network which infers the different percepts and dynamics of the process. Context-specific independence analyses enable us to use the model's structure to directly explore how different contexts affect the way subjects handle the same available information. Hence, we establish that, while the percepts yielded by a unisensory stimulus or by the non-fusion of multisensory stimuli may be similar, they result from different processes, as shown by their differing temporal dynamics. Moreover, our model predicts the impact of bottom-up (stimulus driven) factors as well as of top-down factors (induced by instruction manipulation) on both the perception process and the percept itself.     

Perceptual rivalry: reflexes reveal the gradual nature of visual awareness

Rivalry is a common tool to probe visual awareness: a constant physical stimulus evokes multiple, distinct perceptual interpretations ("percepts") that alternate over time. Percepts are typically described as mutually exclusive, suggesting that a discrete (all-or-none) process underlies changes in visual awareness. Here we follow two strategies to address whether rivalry is an all-or-none process: first, we introduce two reflexes as objective measures of rivalry, pupil dilation and optokinetic nystagmus (OKN); second, we use a continuous input device (analog joystick) to allow observers a gradual subjective report. We find that the "reflexes" reflect the percept rather than the physical stimulus. Both reflexes show a gradual dependence on the time relative to perceptual transitions. Similarly, observers' joystick deflections, which are highly correlated with the reflex measures, indicate gradual transitions. Physically simulating wave-like transitions between percepts suggest piece-meal rivalry (i.e., different regions of space belonging to distinct percepts) as one possible explanation for the gradual transitions. Furthermore, the reflexes show that dominance durations depend on whether or not the percept is actively reported. In addition, reflexes respond to transitions with shorter latencies than the subjective report and show an abundance of short dominance durations. This failure to report fast changes in dominance may result from limited access of introspection to rivalry dynamics. In sum, reflexes reveal that rivalry is a gradual process, rivalry's dynamics is modulated by the required action (response mode), and that rapid transitions in perceptual dominance can slip away from awareness.     

Multi-timescale perceptual history resolves visual ambiguity

When visual input is inconclusive, does previous experience aid the visual system in attaining an accurate perceptual interpretation? Prolonged viewing of a visually ambiguous stimulus causes perception to alternate between conflicting interpretations. When viewed intermittently, however, ambiguous stimuli tend to evoke the same percept on many consecutive presentations. This perceptual stabilization has been suggested to reflect persistence of the most recent percept throughout the blank that separates two presentations. Here we show that the memory trace that causes stabilization reflects not just the latest percept, but perception during a much longer period. That is, the choice between competing percepts at stimulus reappearance is determined by an elaborate history of prior perception. Specifically, we demonstrate a seconds-long influence of the latest percept, as well as a more persistent influence based on the relative proportion of dominance during a preceding period of at least one minute. In case short-term perceptual history and long-term perceptual history are opposed (because perception has recently switched after prolonged stabilization), the long-term influence recovers after the effect of the latest percept has worn off, indicating independence between time scales. We accommodate these results by adding two positive adaptation terms, one with a short time constant and one with a long time constant, to a standard model of perceptual switching.     

Auditory Streaming as an Online Classification Process with Evidence Accumulation

When human subjects hear a sequence of two alternating pure tones, they often perceive it in one of two ways: as one integrated sequence (a single "stream" consisting of the two tones), or as two segregated sequences, one sequence of low tones perceived separately from another sequence of high tones (two "streams"). Perception of this stimulus is thus bistable. Moreover, subjects report on-going switching between the two percepts: unless the frequency separation is large, initial perception tends to be of integration, followed by toggling between integration and segregation phases. The process of stream formation is loosely named “auditory streaming”. Auditory streaming is believed to be a manifestation of human ability to analyze an auditory scene, i.e. to attribute portions of the incoming sound sequence to distinct sound generating entities. Previous studies suggested that the durations of the successive integration and segregation phases are statistically independent. This independence plays an important role in current models of bistability. Contrary to this, we show here, by analyzing a large set of data, that subsequent phase durations are positively correlated. To account together for bistability and positive correlation between subsequent durations, we suggest that streaming is a consequence of an evidence accumulation process. Evidence for segregation is accumulated during the integration phase and vice versa; a switch to the opposite percept occurs stochastically based on this evidence. During a long phase, a large amount of evidence for the opposite percept is accumulated, resulting in a long subsequent phase. In contrast, a short phase is followed by another short phase. We implement these concepts using a probabilistic model that shows both bistability and correlations similar to those observed experimentally.     

Vague-to-crisp dynamics of percept formation modeled as operant (selectionist) process

We model the vague-to-crisp dynamics of forming percepts in the brain by combining two methodologies: dynamic logic (DL) and operant learning process. Forming percepts upon the presentation of visual inputs is likened to model selection based on sampled evidence. Our framework utilizes the DL in selecting the correct “percept” among competing ones, but uses an intrinsic reward mechanism to allow stochastic online update in lieu of performing the optimization step of the DL framework. We discuss the connection of our framework with cognitive processing and the intentional neurodynamic cycle.     

Duality in Binocular Rivalry: Distinct Sensitivity of Percept Sequence and Percept Duration to Imbalance between Monocular Stimuli

Background

Visual perception is usually stable and accurate. However, when the two eyes are simultaneously presented with conflicting stimuli, perception falls into a sequence of spontaneous alternations, switching between one stimulus and the other every few seconds. Known as binocular rivalry, this visual illusion decouples subjective experience from physical stimulation and provides a unique opportunity to study the neural correlates of consciousness. The temporal properties of this alternating perception have been intensively investigated for decades, yet the relationship between two fundamental properties - the sequence of percepts and the duration of each percept - remains largely unexplored.

Methodology/Principal Findings

Here we examine the relationship between the percept sequence and the percept duration by quantifying their sensitivity to the strength imbalance between two monocular stimuli. We found that the percept sequence is far more susceptible to the stimulus imbalance than does the percept duration. The percept sequence always begins with the stronger stimulus, even when the stimulus imbalance is too weak to cause a significant bias in the percept duration. Therefore, introducing a small stimulus imbalance affects the percept sequence, whereas increasing the imbalance affects the percept duration, but not vice versa. To investigate why the percept sequence is so vulnerable to the stimulus imbalance, we further measured the interval between the stimulus onset and the first percept, during which subjects experienced the fusion of two monocular stimuli. We found that this interval is dramatically shortened with increased stimulus imbalance.

Conclusions/Significance

Our study shows that in binocular rivalry, the strength imblanace between monocular stimuli has a much greater impact on the percept sequence than on the percept duration, and increasing this imbalance can accelerate the process responsible for the percept sequence.     

Running as fast as it can: How spiking dynamics form object groupings in the laminar circuits of visual cortex

How spiking neurons cooperate to control behavioral processes is a fundamental problem in computational neuroscience. Such cooperative dynamics are required during visual perception when spatially distributed image fragments are grouped into emergent boundary contours. Perceptual grouping is a challenge for spiking cells because its properties of collinear facilitation and analog sensitivity occur in response to binary spikes with irregular timing across many interacting cells. Some models have demonstrated spiking dynamics in recurrent laminar neocortical circuits, but not how perceptual grouping occurs. Other models have analyzed the fast speed of certain percepts in terms of a single feedforward sweep of activity, but cannot explain other percepts, such as illusory contours, wherein perceptual ambiguity can take hundreds of milliseconds to resolve by integrating multiple spikes over time. The current model reconciles fast feedforward with slower feedback processing, and binary spikes with analog network-level properties, in a laminar cortical network of spiking cells whose emergent properties quantitatively simulate parametric data from neurophysiological experiments, including the formation of illusory contours; the structure of non-classical visual receptive fields; and self-synchronizing gamma oscillations. These laminar dynamics shed new light on how the brain resolves local informational ambiguities through the use of properly designed nonlinear feedback spiking networks which run as fast as they can, given the amount of uncertainty in the data that they process.     

Visual perception of ambiguous figures: synchronization based neural models

We develop and study two neural network models of perceptual alternations. Both models have a star-like architecture of connections with a central element connected to a set of peripheral elements. A particular perception is simulated in terms of partial synchronization between the central element and some sub-group of peripheral elements. The first model is constructed from phase oscillators and the mechanism of perceptual alternations is based on chaotic intermittency under fixed parameter values. Similar to experimental evidence, the distribution of times between perceptual alternations is represented by the gamma distribution. The second model is built of spiking neurons of the Hodgkin–Huxley type. The mechanism of perceptual alternations is based on plasticity of inhibitory synapses which increases the inhibition from the central unit to the neural assembly representing the current percept. As a result another perception is formed. Simulations show that the second model is in good agreement with behavioural data on switching times between percepts of ambiguous figures and with experimental results on binocular rivalry of two and four percepts. This article is part of a special issue on Neuronal Dynamics of Sensory Coding. This special issue is in honour of Professor Pepe Segundo who is one of the pioneers in the study of neural coding. Pepe has been an active participant in many Neural Coding Workshops sharing his great knowledge and experience of research in this field. I (R. Borisyuk) was very happy to meet Pepe for the first time in Prague when attending the first Neural Coding Workshop in 1995. From that time we regularly met at Neural Coding Workshops and these meetings have always been very stimulating and fruitful for my research. Remarkably, the first paper I studied at the beginning of my scientific career was a seminal paper by Moore et al. (1970). For me, this paper provided a great opportunity to learn the basic statistical techniques for the analysis of multiple spike trains and neural coding. According to the Institute of Scientific Information, this paper has been cited 380 times! This exciting paper has inspired my research into the synaptic and functional connectivity of neural circuits derived from spike-train recordings (Borisyuk et al. 1985; Stuart et al. 2005) and guided my search for new ideas on neural coding.     

Atypical Integration of Motion Signals in Autism Spectrum Conditions

Vision in Autism Spectrum Conditions (ASC) is characterized by enhanced perception of local elements, but impaired perception of global percepts. Deficits in coherent motion perception seem to support this characterization, but the roots and robustness of such deficits remain unclear. We aimed to investigate the dynamics of the perceptual decision-making network known to support coherent motion perception. In a series of forced-choice coherent motion perception tests, we parametrically varied a single stimulus dimension, viewing duration, to test whether the rate at which evidence is accumulated towards a global decision is atypical in ASC. 40 adult participants (20 ASC) performed a classic motion discrimination task, manually indicating the global direction of motion in a random-dot kinematogram across a range of coherence levels (2–75%) and stimulus-viewing durations (200–1500 ms). We report a deficit in global motion perception at short viewing durations in ASC. Critically, however, we found that increasing the amount of time over which motion signals could be integrated reduced the magnitude of the deficit, such that at the longest duration there was no difference between the ASC and control groups. Further, the deficit in motion integration at the shortest duration was significantly associated with the severity of autistic symptoms in our clinical population, and was independent from measures of intelligence. These results point to atypical integration of motion signals during the construction of a global percept in ASC. Based on the neural correlates of decision-making in global motion perception our findings suggest the global motion deficit observed in ASC could reflect a slower or more variable response from the primary motion area of the brain or longer accumulation of evidence towards a decision-bound in parietal areas.     

Continuous phase transitions in the perception of multistable visual patterns

The phenomenon of stroboscopic alternative motion exhibits five different percepts that are seen with an increase in the frequency of presentation: (a) succession, (b) fluttering motion, (c) reversible clockwise and counter-clockwise turning motion, (d) oppositional motion and (e) simultaneity. From a synergetic point of view the increase in frequency is a control parameter and the different percepts are order parameters with phase transitions in between. The neural network model of Carmesin and Arndt is applied to receive predictions about hysteresis and phase transitions between these order parameters. Empirical data show the different motion percepts (b), (c) and (e) have lognormal distributions. Following the theoretical model, it is argued that there are three different phases, (a), (c) and (e), with two continuous phase transitions, (b) and (d), between them. The experimental data substantially match the theoretical ssumptions. Received: 29 December 1995 / Accepted in revised form: 3 June 1996     

Opposite influence of perceptual memory on initial and prolonged perception of sensory ambiguity

Observers continually make unconscious inferences about the state of the world based on ambiguous sensory information. This process of perceptual decision-making may be optimized by learning from experience. We investigated the influence of previous perceptual experience on the interpretation of ambiguous visual information. Observers were pre-exposed to a perceptually stabilized sequence of an ambiguous structure-from-motion stimulus by means of intermittent presentation. At the subsequent re-appearance of the same ambiguous stimulus perception was initially biased toward the previously stabilized perceptual interpretation. However, prolonged viewing revealed a bias toward the alternative perceptual interpretation. The prevalence of the alternative percept during ongoing viewing was largely due to increased durations of this percept, as there was no reliable decrease in the durations of the pre-exposed percept. Moreover, the duration of the alternative percept was modulated by the specific characteristics of the pre-exposure, whereas the durations of the pre-exposed percept were not. The increase in duration of the alternative percept was larger when the pre-exposure had lasted longer and was larger after ambiguous pre-exposure than after unambiguous pre-exposure. Using a binocular rivalry stimulus we found analogous perceptual biases, while pre-exposure did not affect eye-bias. We conclude that previously perceived interpretations dominate at the onset of ambiguous sensory information, whereas alternative interpretations dominate prolonged viewing. Thus, at first instance ambiguous information seems to be judged using familiar percepts, while re-evaluation later on allows for alternative interpretations.     

Visible Persistence of Single-Transient Random Dot Patterns: Spatial Parameters Affect the Duration of Fading Percepts

Visible persistence refers to the continuation of visual perception after the physical termination of a stimulus. We studied an extreme case of visible persistence by presenting two matrices of randomly distributed black and white pixels in succession. On the transition from one matrix to the second, the luminance polarity of all pixels within a disk- or annulus-shaped area reversed, physically creating a single second-order transient signal. This transient signal produces the percept of a disk or an annulus with an abrupt onset and a gradual offset. To study the nature of this fading percept we varied spatial parameters, such as the inner and the outer diameter of annuli (Experiment I) and the radius and eccentricity of disks (Experiment III), and measured the duration of visible persistence by having subjects adjust the synchrony of the onset of a reference stimulus with the onset or the offset of the fading percept. We validated this method by comparing two modalities of the reference stimuli (Experiment I) and by comparing the judgments of fading percepts with the judgments of stimuli that actually fade in luminance contrast (Experiment II). The results show that (i) irrespective of the reference modality, participants are able to precisely judge the on- and the offsets of the fading percepts, (ii) auditory reference stimuli lead to higher visible persistence durations than visual ones, (iii) visible persistence duration increases with the thickness of annuli and the diameter of disks, but decreases with the diameter of annuli, irrespective of stimulus eccentricity. These effects cannot be explained by stimulus energy, which suggests that more complex processing mechanisms are involved. Seemingly contradictory effects of disk and annulus diameter can be unified by assuming an abstract filling-in mechanism that speeds up with the strength of the edge signal and takes more time the larger the stimulus area is.     

Depth perception from pairs of overlapping cues in pictorial displays

The experiments reported herein probe the visual cortical mechanisms that control near-far percepts in response to two-dimensional stimuli. Figural contrast is found to be a principal factor for the emergence of percepts of near versus far in pictorial stimuli, especially when stimulus duration is brief. Pictorial factors such as interposition (Experiment 1) and partial occlusion (Experiments 2 and 3) may cooperate, as generally predicted by cue combination models, or compete with contrast factors in the manner predicted by the FACADE model. In particular, if the geometrical configuration of an image favors activation of cortical bipole grouping cells, as at the top of a T-junction, then this advantage can cooperate with the contrast of the configuration to facilitate a near-far percept at a lower contrast than at an X-junction. Varying the exposure duration of the stimuli shows that the more balanced bipole competition in the X-junction case takes longer exposures to resolve than the bipole competition in the T-junction case (Experiment 3).     

New strategies for improving speech enhancement

We describe two design strategies that could substantially improve the performance of speech enhancement systems. Results from a preliminary study of pulse recovery are presented to illustrate the potential benefits of such strategies. The first strategy is a direct application of a non-linear, adaptive signal processing approach for recovery of speech in noise. The second strategy optimizes performance by maximizing the enhancement system's ability to evoke target speech percepts. This approach may lead to better performance because the design is optimized on a measure directly related to the ultimate goal of speech enhancement: accurate communication of the speech percept. In both systems, recently developed ‘neural network’ learning algorithms can be used to determine appropriate parameters for enhancement processing.     

Dynamical vs. judgmental comparison: hysteresis effects in motion perception

Perceptual comparison was investigated by gradually varying the relative length of two apparent motion paths, and independently determining when an initial percept was lost during the course of attribute change and when an alternative percept emerged. Dynamical comparison was indicated by a range of attribute values for which perception was bistable. Within this range, a percept that lost stability was immediately replaced by an alternative percept. Judgmental comparison was indicated by a range of attribute values for which perception was uncertain. When an initial percept was lost, an alternative percept did not immediately emerge because the alternatives being compared could not be distinguished. Differences in the effects of random noise on dynamical vs. judgmental comparison were demonstrated with computational simulations, and implications are discussed for motion energy models and solutions to the motion correspondence problem.     

Beta,but Not Gamma,Band Oscillations Index Visual Form-Motion Integration

Electrophysiological oscillations in different frequency bands co-occur with perceptual, motor and cognitive processes but their function and respective contributions to these processes need further investigations. Here, we recorded MEG signals and seek for percept related modulations of alpha, beta and gamma band activity during a perceptual form/motion integration task. Participants reported their bound or unbound perception of ambiguously moving displays that could either be seen as a whole square-like shape moving along a Lissajou''s figure (bound percept) or as pairs of bars oscillating independently along cardinal axes (unbound percept). We found that beta (15–25 Hz), but not gamma (55–85 Hz) oscillations, index perceptual states at the individual and group level. The gamma band activity found in the occipital lobe, although significantly higher during visual stimulation than during base line, is similar in all perceptual states. Similarly, decreased alpha activity during visual stimulation is not different for the different percepts. Trial-by-trial classification of perceptual reports based on beta band oscillations was significant in most observers, further supporting the view that modulation of beta power reliably index perceptual integration of form/motion stimuli, even at the individual level.     

A continuous-time neural model for sequential action

Action selection, planning and execution are continuous processes that evolve over time, responding to perceptual feedback as well as evolving top-down constraints. Existing models of routine sequential action (e.g. coffee- or pancake-making) generally fall into one of two classes: hierarchical models that include hand-built task representations, or heterarchical models that must learn to represent hierarchy via temporal context, but thus far lack goal-orientedness. We present a biologically motivated model of the latter class that, because it is situated in the Leabra neural architecture, affords an opportunity to include both unsupervised and goal-directed learning mechanisms. Moreover, we embed this neurocomputational model in the theoretical framework of the theory of event coding (TEC), which posits that actions and perceptions share a common representation with bidirectional associations between the two. Thus, in this view, not only does perception select actions (along with task context), but actions are also used to generate perceptions (i.e. intended effects). We propose a neural model that implements TEC to carry out sequential action control in hierarchically structured tasks such as coffee-making. Unlike traditional feedforward discrete-time neural network models, which use static percepts to generate static outputs, our biological model accepts continuous-time inputs and likewise generates non-stationary outputs, making short-timescale dynamic predictions.     

An auditory neural correlate suggests a mechanism underlying holistic pitch perception

Current theories of auditory pitch perception propose that cochlear place (spectral) and activity timing pattern (temporal) information are somehow combined within the brain to produce holistic pitch percepts, yet the neural mechanisms for integrating these two kinds of information remain obscure. To examine this process in more detail, stimuli made up of three pure tones whose components are individually resolved by the peripheral auditory system, but that nonetheless elicit a holistic, "missing fundamental" pitch percept, were played to human listeners. A technique was used to separate neural timing activity related to individual components of the tone complexes from timing activity related to an emergent feature of the complex (the envelope), and the region of the tonotopic map where information could originate from was simultaneously restricted by masking noise. Pitch percepts were mirrored to a very high degree by a simple combination of component-related and envelope-related neural responses with similar timing that originate within higher-frequency regions of the tonotopic map where stimulus components interact. These results suggest a coding scheme for holistic pitches whereby limited regions of the tonotopic map (spectral places) carrying envelope- and component-related activity with similar timing patterns selectively provide a key source of neural pitch information. A similar mechanism of integration between local and emergent object properties may contribute to holistic percepts in a variety of sensory systems.     

The cutaneous rabbit illusion affects human primary sensory cortex somatotopically

We used functional magnetic resonance imaging (fMRI) to study neural correlates of a robust somatosensory illusion that can dissociate tactile perception from physical stimulation. Repeated rapid stimulation at the wrist, then near the elbow, can create the illusion of touches at intervening locations along the arm, as if a rabbit hopped along it. We examined brain activity in humans using fMRI, with improved spatial resolution, during this version of the classic cutaneous rabbit illusion. As compared with control stimulation at the same skin sites (but in a different order that did not induce the illusion), illusory sequences activated contralateral primary somatosensory cortex, at a somatotopic location corresponding to the filled-in illusory perception on the forearm. Moreover, the amplitude of this somatosensory activation was comparable to that for veridical stimulation including the intervening position on the arm. The illusion additionally activated areas of premotor and prefrontal cortex. These results provide direct evidence that illusory somatosensory percepts can affect primary somatosensory cortex in a manner that corresponds somatotopically to the illusory percept.     

Do variables that affect similar bistable apparent-movement displays result in similar changes in perception?

Two bistable apparent-movement displays (i.e. ones that generate two qualitatively different kinds of movement percepts under different conditions) were compared. They were designed to be as similar as possible spatially, and were studied with identical stimulus manipulations to see whether changes in balance between their bistable percepts would be similar. Results show that the two displays had different response characteristics to the same stimulus manipulations. Two models of motion perception that have previously predicted at least one kind of bistable apparent motion were considered in terms of how well they address the current data. As yet, neither model has been shown to predict the motion states and bistable behavior of the two displays studied here. It is concluded that results of the type described here (specifically, differences in the psychophysical functions yielded by two structurally similar but qualitatively different bistable displays) present a challenge for theories of motion perception.