Predicting human perceptual decisions by decoding neuronal information profiles

Perception relies on the response of populations of neurons in sensory cortex. How the response profile of a neuronal population gives rise to perception and perceptual discrimination has been conceptualized in various ways. Here we suggest that neuronal population responses represent information about our environment explicitly as Fisher information (FI), which is a local measure of the variance estimate of the sensory input. We show how this sensory information can be read out and combined to infer from the available information profile which stimulus value is perceived during a fine discrimination task. In particular, we propose that the perceived stimulus corresponds to the stimulus value that leads to the same information for each of the alternative directions, and compare the model prediction to standard models considered in the literature (population vector, maximum likelihood, maximum-a-posteriori Bayesian inference). The models are applied to human performance in a motion discrimination task that induces perceptual misjudgements of a target direction of motion by task irrelevant motion in the spatial surround of the target stimulus (motion repulsion). By using the neurophysiological insight that surround motion suppresses neuronal responses to the target motion in the center, all models predicted the pattern of perceptual misjudgements. The variation of discrimination thresholds (error on the perceived value) was also explained through the changes of the total FI content with varying surround motion directions. The proposed FI decoding scheme incorporates recent neurophysiological evidence from macaque visual cortex showing that perceptual decisions do not rely on the most active neurons, but rather on the most informative neuronal responses. We statistically compare the prediction capability of the FI decoding approach and the standard decoding models. Notably, all models reproduced the variation of the perceived stimulus values for different surrounds, but with different neuronal tuning characteristics underlying perception. Compared to the FI approach the prediction power of the standard models was based on neurons with far wider tuning width and stronger surround suppression. Our study demonstrates that perceptual misjudgements can be based on neuronal populations encoding explicitly the available sensory information, and provides testable neurophysiological predictions on neuronal tuning characteristics underlying human perceptual decisions.     

Phase-linking and the perceived motion during off-vertical axis rotation

Human off-vertical axis rotation (OVAR) in the dark typically produces perceived motion about a cone, the amplitude of which changes as a function of frequency. This perception is commonly attributed to the fact that both the OVAR and the conical motion have a gravity vector that rotates about the subject. Little-known, however, is that this rotating-gravity explanation for perceived conical motion is inconsistent with basic observations about self-motion perception: (a) that the perceived vertical moves toward alignment with the gravito-inertial acceleration (GIA) and (b) that perceived translation arises from perceived linear acceleration, as derived from the portion of the GIA not associated with gravity. Mathematically proved in this article is the fact that during OVAR these properties imply mismatched phase of perceived tilt and translation, in contrast to the common perception of matched phases which correspond to conical motion with pivot at the bottom. This result demonstrates that an additional perceptual rule is required to explain perception in OVAR. This study investigates, both analytically and computationally, the phase relationship between tilt and translation at different stimulus rates—slow (45°/s) and fast (180°/s), and the three-dimensional shape of predicted perceived motion, under different sets of hypotheses about self-motion perception. We propose that for human motion perception, there is a phase-linking of tilt and translation movements to construct a perception of one’s overall motion path. Alternative hypotheses to achieve the phase match were tested with three-dimensional computational models, comparing the output with published experimental reports. The best fit with experimental data was the hypothesis that the phase of perceived translation was linked to perceived tilt, while the perceived tilt was determined by the GIA. This hypothesis successfully predicted the bottom-pivot cone commonly reported and a reduced sense of tilt during fast OVAR. Similar considerations apply to the hilltop illusion often reported during horizontal linear oscillation. Known response properties of central neurons are consistent with this ability to phase-link translation with tilt. In addition, the competing “standard” model was mathematically proved to be unable to predict the bottom-pivot cone regardless of the values used for parameters in the model.     

Visual clutter causes high-magnitude errors

Perceptual decisions are often made in cluttered environments, where a target may be confounded with competing “distractor” stimuli. Although many studies and theoretical treatments have highlighted the effect of distractors on performance, it remains unclear how they affect thequality of perceptual decisions. Here we show that perceptual clutter leads not only to an increase in judgment errors, but also to an increase in perceived signal strength and decision confidence on erroneous trials. Observers reported simultaneously the direction and magnitude of the tilt of a target grating presented either alone, or together with vertical distractor stimuli. When presented in isolation, observers perceived isolated targets as only slightly tilted on error trials, and had little confidence in their decision. When the target was embedded in distractors, however, they perceived it to be strongly tilted on error trials, and had high confidence of their (erroneous) decisions. The results are well explained by assuming that the observers' internal representation of stimulus orientation arises from a nonlinear combination of the outputs of independent noise-perturbed front-end detectors. The implication that erroneous perceptual decisions in cluttered environments are made with high confidence has many potential practical consequences, and may be extendable to decision-making in general.     

Computational modelling of motion detectors: responses to two-frame displays

Schemes for motion detection fall into two classes. Reichardt correlators compare spatial luminance patterns at two locations at different times; gradient detectors compare spatial and temporal luminance gradients. Both are candidate operators for biological and machine vision systems. A large body of perceptual data exists, defining the properties of motion detectors used by human observers, which can form a basis for determining which class of detector is appropriate for the human visual system. Plausible versions of each detector were implemented, and their responses to a variety of two-frame stimuli were computed. Results indicated that both detectors can predict most of the data, but on balance gradient detectors offer the best working hypothesis for motion detection by human observers. This conclusion is necessarily limited to the type of stimuli used, and may require modification in the light of responses to continuously moving stimuli.     

A motion-dependent distortion of retinotopy in area V4

When one element in an apparent motion sequence differs in color from the others, it is perceived as shifted along the motion trajectory. We examined whether V4 neurons encode the physical or perceived location of this "flashed" element by recording neuronal responses while monkeys viewed these stimuli. The retinotopic locus of V4 activity evoked by the flashed element shifted along the motion trajectory. The magnitude of the shift is consistent with the perceptual shift in humans viewing identical stimuli. This retinotopic distortion depended on the presence of a flashed element but was observed for both color-selective and non-color-selective neurons. The distortion was undiminished when the flashed element terminated the sequence, a condition that reduced the perceptual shift in humans. These findings are consistent with a Bayesian model of localization in which perceived location is derived from position signals optimally integrated across visual areas.     

A Laterally Interconnected Neural Architecture in MST Accounts for Psychophysical Discrimination of Complex Motion Patterns

The complex patterns of visual motion formed across the retina during self-motion, often referred to as optic flow, provide a rich source of information describing our dynamic relationship within the environment. Psychophysical studies indicate the existence of specialized detectors for component motion patterns (radial, circular, planar) that are consistent with the visual motion properties of cells in the medial superior temporal area (MST) of nonhuman primates. Here we use computational modeling and psychophysics to investigate the structural and functional role of these specialized detectors in performing a graded motion pattern (GMP) discrimination task. In the psychophysical task perceptual discrimination varied significantly with the type of motion pattern presented, suggesting perceptual correlates to the preferred motion bias reported in MST. Simulated perceptual discrimination in a population of independent MST-like neural responses showed inconsistent psychophysical performance that varied as a function of the visual motion properties within the population code. Robust psychophysical performance was achieved by fully interconnecting neural populations such that they inhibited nonpreferred units. Taken together, these results suggest that robust processing of the complex motion patterns associated with self-motion and optic flow may be mediated by an inhibitory structure of neural interactions in MST.     

A motion illusion reveals mechanisms of perceptual stabilization

Visual illusions are valuable tools for the scientific examination of the mechanisms underlying perception. In the peripheral drift illusion special drift patterns appear to move although they are static. During fixation small involuntary eye movements generate retinal image slips which need to be suppressed for stable perception. Here we show that the peripheral drift illusion reveals the mechanisms of perceptual stabilization associated with these micromovements. In a series of experiments we found that illusory motion was only observed in the peripheral visual field. The strength of illusory motion varied with the degree of micromovements. However, drift patterns presented in the central (but not the peripheral) visual field modulated the strength of illusory peripheral motion. Moreover, although central drift patterns were not perceived as moving, they elicited illusory motion of neutral peripheral patterns. Central drift patterns modulated illusory peripheral motion even when micromovements remained constant. Interestingly, perceptual stabilization was only affected by static drift patterns, but not by real motion signals. Our findings suggest that perceptual instabilities caused by fixational eye movements are corrected by a mechanism that relies on visual rather than extraretinal (proprioceptive or motor) signals, and that drift patterns systematically bias this compensatory mechanism. These mechanisms may be revealed by utilizing static visual patterns that give rise to the peripheral drift illusion, but remain undetected with other patterns. Accordingly, the peripheral drift illusion is of unique value for examining processes of perceptual stabilization.     

Functional segregation and temporal hierarchy of the visual perceptive systems.

In extending our previous work, we addressed the question of whether different visual attributes are perceived separately when they belong to different objects, rather than the same one. Using our earlier psychophysical method, but separating the attributes to be paired in two different halves of the screen, we found that human subjects misbind the colour and the direction of motion, or the colour and the orientation of lines, because colour, form, and motion are perceived separately and at different times. The results therefore show that there is a perceptual temporal hierarchy in vision.     

Asynchronous processing in vision: color leads motion

It has been demonstrated that subjects do not report changes in color and direction of motion as being co-incidental when they occur synchronously. Instead, for the changes to be reported as being synchronous, changes in direction of motion must precede changes in color. To explain this observation, some researchers have suggested that the neural processing of color and motion is asynchronous. This interpretation has been criticized on the basis that processing time may not correlate directly and invariantly with perceived time of occurrence. Here we examine this possibility by making use of the color-contingent motion aftereffect. By correlating color states disproportionately with two directions of motion, we produced and measured color-contingent motion aftereffects as a function of the range of physical correlations. The aftereffects observed are consistent with the perceptual correlation between color and motion being different from the physical correlation. These findings demonstrate asynchronous processing for different stimulus attributes, with color being processed more quickly than motion. This suggests that the time course of perceptual experience correlates directly with that of neural activity.     

Transitions between central and peripheral vision create spatial/temporal distortions: a hypothesis concerning the perceived break of the curveball

Background

The human visual system does not treat all parts of an image equally: the central segments of an image, which fall on the fovea, are processed with a higher resolution than the segments that fall in the visual periphery. Even though the differences between foveal and peripheral resolution are large, these differences do not usually disrupt our perception of seamless visual space. Here we examine a motion stimulus in which the shift from foveal to peripheral viewing creates a dramatic spatial/temporal discontinuity.

Methodology/Principal Findings

The stimulus consists of a descending disk (global motion) with an internal moving grating (local motion). When observers view the disk centrally, they perceive both global and local motion (i.e., observers see the disk''s vertical descent and the internal spinning). When observers view the disk peripherally, the internal portion appears stationary, and the disk appears to descend at an angle. The angle of perceived descent increases as the observer views the stimulus from further in the periphery. We examine the first- and second-order information content in the display with the use of a three-dimensional Fourier analysis and show how our results can be used to describe perceived spatial/temporal discontinuities in real-world situations.

Conclusions/Significance

The perceived shift of the disk''s direction in the periphery is consistent with a model in which foveal processing separates first- and second-order motion information while peripheral processing integrates first- and second-order motion information. We argue that the perceived distortion may influence real-world visual observations. To this end, we present a hypothesis and analysis of the perception of the curveball and rising fastball in the sport of baseball. The curveball is a physically measurable phenomenon: the imbalance of forces created by the ball''s spin causes the ball to deviate from a straight line and to follow a smooth parabolic path. However, the curveball is also a perceptual puzzle because batters often report that the flight of the ball undergoes a dramatic and nearly discontinuous shift in position as the ball nears home plate. We suggest that the perception of a discontinuous shift in position results from differences between foveal and peripheral processing.     

How do painters represent motion in garments? Graphic invariants across centuries

Western visual art has radically changed throughout the centuries: different techniques, interest in the representation of reality, and use of graphic signs. Indeed, only a few pictorial cues have retained the same meaning and use. These kinds of graphic invariants may play a key role not only in a comparative study of art history, but also for discovering underlying common perceptual mechanisms. Here the aim is to show that western painters use the same graphic solutions to represent motion in garments, across countries and centuries. A pilot experiment, using 160 paintings representative of all main western European art movements from the thirteenth to the twentieth century, shows that different artists represented the motion of garments with the same orientation, curvature and convergence of lines. Experiment 1 demonstrates, with a smaller sample of paintings (16, i.e. two per century) that the relationship between orientation, curvature and convergence of lines is a good predictor of perceived motion. Experiment 2 shows how the same garments, isolated from the context of the paintings, still give different dynamic impressions according to the same rules. Finally, Experiment 3 confirmed the same results, whilst patterns previously used are simplified to their geometrical structure. These results call for an underlying perceptual mechanism that specifically recognizes orientation, curvature and parallelism levels as cues of motion in a static pattern.     

Speed encoding in correlation motion detectors as a consequence of spatial structure

For animals to carry out a wide range of detection, recognition and navigation tasks, visual motion signals are crucial. The encoding of motion information has therefore, attracted much attention in the experimental and computational study of brain function. Two main alternative mechanisms have been proposed on the basis of behavioural and physiological experiments. On one hand, correlation-type and motion energy detectors are simple and efficient in the design of their basic mechanism but are tuned to temporal frequency rather than to speed. On other hand, gradient-type motion detectors directly represent an estimate of speed, but may require more demanding processing mechanisms. We demonstrate here how the temporal frequency dependence observed for sine-wave gratings can disappear for less constrained stimuli, to be replaced by responses reflecting speed for stimuli like square waves when a phase-sensitive detection mechanism is employed. We conclude from these observations that temporal frequency tuning is not necessarily a limitation for motion vision based on correlation detectors, and more generally demonstrate in view of the typical Fourier composition of natural scenes, that correlation detectors operating in such environments can encode image speed. In the context of our results, we discuss the implications of the loss of phase sensitivity inherent in using a linear system approach to describe neural processing.     

Seeing objects in motion

This paper reports estimates of the conjoint spatiotemporal tuning functions of the neural mechanisms of the human vision system which detect image motion. The functions were derived from measurements of the minimum contrast necessary to detect the direction of drift of a sinusoidal grating, in the presence of phase-reversed masking gratings of various spatial and temporal frequencies. A mask of similar spatial and temporal frequencies to the test grating reduces sensitivity considerably, whereas one differing greatly in spatial or temporal frequency has little or no effect. The results show that for test gratings drifting at 8 Hz, the tuning function is bandpass in both space and time, peaked at the temporal and spatial frequency (SF) of the test (SFs were 0.1, 1 or 5 c deg-1; c represents cycles throughout). For a grating of 5 c deg-1 drifting at 0.3 Hz, the function is bandpass in space but lowpass in time. Fourier transform of the frequency results yields a function in space-time which we term the 'spatiotemporal receptive field'. For movement detectors (bandpass in space and time) the fields comprise alternating ridges of opposing polarity, elongated in space-time along the preferred velocity axis of the detector. We suggest that this organization explains how detectors analyse form and motion concurrently and accounts, at least in part, for a variety of perceptual phenomena, including summation, reduction of motion smear, metacontrast, stroboscopic motion and spatiotemporal interpolation.     

Evidence for perceptual "trapping" and adaptation in multistable binocular rivalry

When a different pattern is presented to each eye, the perceived image spontaneously alternates between the two patterns (binocular rivalry); the dynamics of these bistable alternations are known to be stochastic. Examining multistable binocular rivalry (involving four dominant percepts), we demonstrated path dependence and on-line adaptation, which were equivalent whether perceived patterns were formed by single-eye dominance or by mixed-eye dominance. The spontaneous perceptual transitions tended to get trapped within a pair of related global patterns (e.g., opponent shapes and symmetric patterns), and during such trapping, the probability of returning to the repeatedly experienced patterns gradually decreased (postselection pattern adaptation). These results suggest that the structure of global shape coding and its adaptation play a critical role in directing spontaneous alternations of visual awareness in perceptual multistability.     

When motion is not perceived: evidence from adaptation and dynamical stability

Adaptation was used to probe the perceiver's activation state when either motion or nonmotion percepts are formed for bistable, single-element apparent motion stimuli. Although adaptation was not observed in every instance, when it was observed its effect was to increase the probability of both motion-to-nonmotion and nonmotion-to-motion switches, the time scale of adaptation corresponding to neurophysiological observations for directionally selective cortical cells (Giaschi et al. 1993). This susceptibility to de-stabilizing adaptation effects indicated that the nonmotion percept was not the result of inadequate stimulation producing subthreshold levels of motion detector activation; if that were the case, activation-dependent adaptation would have decreased the nonmotion-to-motion switching rate by reducing activation further below threshold. Above-threshold activation levels are therefore associated with both nonmotion and motion perceptual states, and the failure to perceive motion despite the presence of adequate motion detector stimulation can be attributed to inhibitory competition between detectors activated by motion-specifying stimulus information and detectors activated to similar levels by motion-independent stimulus information, consistent with the dynamical quality of single-element apparent motion.     

The effects of task and saliency on latencies for colour and motion processing

In human visual perception, there is evidence that different visual attributes, such as colour, form and motion, have different neural-processing latencies. Specifically, recent studies have suggested that colour changes are processed faster than motion changes. We propose that the processing latencies should not be considered as fixed quantities for different attributes, but instead depend upon attribute salience and the observer's task. We asked observers to respond to high- and low-salience colour and motion changes in three different tasks. The tasks varied from having a strong motor component to having a strong perceptual component. Increasing salience led to shorter processing times in all three tasks. We also found an interaction between task and attribute: motion was processed more quickly in reaction-time tasks, whereas colour was processed more quickly in more perceptual tasks. Our results caution against making direct comparisons between latencies for processing different visual attributes without equating salience or considering task effects. More-salient attributes are processed faster than less-salient ones, and attributes that are critical for the task are also processed more quickly.     

Perceptual grouping of biological motion promotes binocular rivalry

Investigation of perceptual rivalry between conflicting stimuli presented one to each eye can further understanding of the neural underpinnings of conscious visual perception. During rivalry, visual awareness fluctuates between perceptions of the two stimuli. Here, we demonstrate that high-level perceptual grouping can promote rivalry between stimulus pairs that would otherwise be perceived as nonrivalrous. Perceptual grouping was generated with point-light walker stimuli that simulate human motion, visible only as lights placed on the joints. Although such walking figures are unrecognizable when stationary, recognition judgments as complex as gender and identity can accurately be made from animated displays, demonstrating the efficiency with which our visual system can group dynamic local signals into a globally coherent walking figure. We find that point-light walker stimuli presented one to each eye and in different colors and configurations results in strong rivalry. However, rivalry is minimal when the two walkers are split between the eyes or both presented to one eye. This pattern of results suggests that processing animated walker figures promotes rivalry between signals from the two eyes rather than between higher-level representations of the walkers. This leads us to hypothesize that awareness during binocular rivalry involves the integrated activity of high-level perceptual mechanisms in conjunction with lower-level ocular suppression modulated via cortical feedback.     

On perceptual analyzers underlying visual texture discrimination: Part I

We have found a class of feature detectors, based on the quasi-collinearity of dots, which result in visual texture discrimination even when second order statistics are equal. This degenerate counterexample to the Julesz conjecture on effortless texture discrimination has supplied the key to a simple theory of texture discrimination. Accordingly, effortless texture discrimination is based on two classses of perceptual detectors: Class A, those that measure differences in second-order (dipole) statistics; Class B, those that can still detect statistical differences in some features when second-order statistics are kept identical; for instance, the quasi-collinearity of adjacent dipoles. The difference thresholds (tuning curves) for the perceptual dipole and quasi-collinearity detectors have been determined. These texture pairs were generated by a method that creates micropatterns with iso-dipole duals from 4 disks. The extension of this 4-disk method to 5 and more disks with iso-dipole duals permits the search for other kinds of perceptual detectors and will be discussed in Part II.     

Onset rivalry: brief presentation isolates an early independent phase of perceptual competition

When the left and right eyes are simultaneously presented with different images, observers typically report exclusive awareness of only one image. This phenomenon is termed binocular rivalry, reflecting the fact that the dominant image alternates every few seconds in a cycle of perceptual competition that continues indefinitely. Despite the apparent continuity in perceptual switching, we now demonstrate that the initial "onset" period is fundamentally different to all subsequent rivalry epochs. Using brief intermittent presentations, rivalry dominance shows strong biases such that the same target is perceived with each successive stimulus onset. These biases remain consistent within any given location, but vary across the visual field in a distribution that is stable over multiple weeks but highly idiosyncratic across observers. If the presentation exceeds approximately 1sec at any location, however, the very different and much more balanced alternations of sustained binocular rivalry become apparent. These powerful onset biases are observed with brief intermittent presentations at a single location or with continual smooth motion of the targets. Periods of adaptation to one of the rivaling targets induced local switches in dominance to the non-adapted target. However, these effects were generally limited to the spatial site of adaptation and had less influence over each subsequent cycle of the target. We conclude that onset rivalry is independent of sustained rivalry and cannot be explained by local regions of monocular dominance or memory of past perceptual history, but rather reflects low-level, spatially localized factors that are stable over periods of weeks. These findings suggest that brief presentation paradigms are inappropriate for their current use in studies of the mechanisms underlying sustained rivalry. However, brief presentations are ideal for investigating early stages of perceptual competition.     

Visual Working Memory Contents Bias Ambiguous Structure from Motion Perception

The way we perceive the visual world depends crucially on the state of the observer. In the present study we show that what we are holding in working memory (WM) can bias the way we perceive ambiguous structure from motion stimuli. Holding in memory the percept of an unambiguously rotating sphere influenced the perceived direction of motion of an ambiguously rotating sphere presented shortly thereafter. In particular, we found a systematic difference between congruent dominance periods where the perceived direction of the ambiguous stimulus corresponded to the direction of the unambiguous one and incongruent dominance periods. Congruent dominance periods were more frequent when participants memorized the speed of the unambiguous sphere for delayed discrimination than when they performed an immediate judgment on a change in its speed. The analysis of dominance time-course showed that a sustained tendency to perceive the same direction of motion as the prior stimulus emerged only in the WM condition, whereas in the attention condition perceptual dominance dropped to chance levels at the end of the trial. The results are explained in terms of a direct involvement of early visual areas in the active representation of visual motion in WM.