Explaining the Timing of Natural Scene Understanding with a Computational Model of Perceptual Categorization

Observers can rapidly perform a variety of visual tasks such as categorizing a scene as open, as outdoor, or as a beach. Although we know that different tasks are typically associated with systematic differences in behavioral responses, to date, little is known about the underlying mechanisms. Here, we implemented a single integrated paradigm that links perceptual processes with categorization processes. Using a large image database of natural scenes, we trained machine-learning classifiers to derive quantitative measures of task-specific perceptual discriminability based on the distance between individual images and different categorization boundaries. We showed that the resulting discriminability measure accurately predicts variations in behavioral responses across categorization tasks and stimulus sets. We further used the model to design an experiment, which challenged previous interpretations of the so-called “superordinate advantage.” Overall, our study suggests that observed differences in behavioral responses across rapid categorization tasks reflect natural variations in perceptual discriminability.     

Similarity of Cortical Activity Patterns Predicts generalization Behavior

Humans and animals readily generalize previously learned knowledge to new situations. Determining similarity is critical for assigning category membership to a novel stimulus. We tested the hypothesis that category membership is initially encoded by the similarity of the activity pattern evoked by a novel stimulus to the patterns from known categories. We provide behavioral and neurophysiological evidence that activity patterns in primary auditory cortex contain sufficient information to explain behavioral categorization of novel speech sounds by rats. Our results suggest that category membership might be encoded by the similarity of the activity pattern evoked by a novel speech sound to the patterns evoked by known sounds. Categorization based on featureless pattern matching may represent a general neural mechanism for ensuring accurate generalization across sensory and cognitive systems.     

Good Exemplars of Natural Scene Categories Elicit Clearer Patterns than Bad Exemplars but Not Greater BOLD Activity

Within the range of images that we might categorize as a “beach”, for example, some will be more representative of that category than others. Here we first confirmed that humans could categorize “good” exemplars better than “bad” exemplars of six scene categories and then explored whether brain regions previously implicated in natural scene categorization showed a similar sensitivity to how well an image exemplifies a category. In a behavioral experiment participants were more accurate and faster at categorizing good than bad exemplars of natural scenes. In an fMRI experiment participants passively viewed blocks of good or bad exemplars from the same six categories. A multi-voxel pattern classifier trained to discriminate among category blocks showed higher decoding accuracy for good than bad exemplars in the PPA, RSC and V1. This difference in decoding accuracy cannot be explained by differences in overall BOLD signal, as average BOLD activity was either equivalent or higher for bad than good scenes in these areas. These results provide further evidence that V1, RSC and the PPA not only contain information relevant for natural scene categorization, but their activity patterns mirror the fundamentally graded nature of human categories. Analysis of the image statistics of our good and bad exemplars shows that variability in low-level features and image structure is higher among bad than good exemplars. A simulation of our neuroimaging experiment suggests that such a difference in variance could account for the observed differences in decoding accuracy. These results are consistent with both low-level models of scene categorization and models that build categories around a prototype.     

Shaping of Receptive Fields in the Visual Cortex During Retinal Maturation

We present a computational study of the formation of simple-cell receptive field patterns in the primary visual cortex. Based on the observation that the spatial frequency of the retinal filter increases postnatally, our results explain differences in the time course of the development of orientation selectivity in binocularly deprived and normally reared kittens. Development after eye-opening in normal animals is modelled by training with natural images, whereas in the case of binocular deprivation noise-like stimulation continues. Further, it is shown that different orientation selectivities are obtained for network models trained with natural images in contrast to random phase images of identical second order statistics. The latter finding suggests that higher-order statistics of the inputs influences development of primary visual cortex. Finally, we search for quantities that identify possible signatures of natural image statistics in order to specify the amount of constructiveness that visual experience has on the formation of receptive fields.     

The biological basis of a universal constraint on color naming: cone contrasts and the two-way categorization of colors

Many studies have provided evidence for the existence of universal constraints on color categorization or naming in various languages, but the biological basis of these constraints is unknown. A recent study of the pattern of color categorization across numerous languages has suggested that these patterns tend to avoid straddling a region in color space at or near the border between the English composite categories of "warm" and "cool". This fault line in color space represents a fundamental constraint on color naming. Here we report that the two-way categorization along the fault line is correlated with the sign of the L- versus M-cone contrast of a stimulus color. Moreover, we found that the sign of the L-M cone contrast also accounted for the two-way clustering of the spatially distributed neural responses in small regions of the macaque primary visual cortex, visualized with optical imaging. These small regions correspond to the hue maps, where our previous study found a spatially organized representation of stimulus hue. Altogether, these results establish a direct link between a universal constraint on color naming and the cone-specific information that is represented in the primate early visual system.     

Neural correlates of face and object perception in an awake chimpanzee (Pan troglodytes) examined by scalp-surface event-related potentials

Background

The neural system of our closest living relative, the chimpanzee, is a topic of increasing research interest. However, electrophysiological examinations of neural activity during visual processing in awake chimpanzees are currently lacking.

Methodology/Principal Findings

In the present report, skin-surface event-related brain potentials (ERPs) were measured while a fully awake chimpanzee observed photographs of faces and objects in two experiments. In Experiment 1, human faces and stimuli composed of scrambled face images were displayed. In Experiment 2, three types of pictures (faces, flowers, and cars) were presented. The waveforms evoked by face stimuli were distinguished from other stimulus types, as reflected by an enhanced early positivity appearing before 200 ms post stimulus, and an enhanced late negativity after 200 ms, around posterior and occipito-temporal sites. Face-sensitive activity was clearly observed in both experiments. However, in contrast to the robustly observed face-evoked N170 component in humans, we found that faces did not elicit a peak in the latency range of 150–200 ms in either experiment.

Conclusions/Significance

Although this pilot study examined a single subject and requires further examination, the observed scalp voltage patterns suggest that selective processing of faces in the chimpanzee brain can be detected by recording surface ERPs. In addition, this non-invasive method for examining an awake chimpanzee can be used to extend our knowledge of the characteristics of visual cognition in other primate species.     

Temporal Dynamics of Visual Attention Measured with Event-Related Potentials

How attentional modulation on brain activities determines behavioral performance has been one of the most important issues in cognitive neuroscience. This issue has been addressed by comparing the temporal relationship between attentional modulations on neural activities and behavior. Our previous study measured the time course of attention with amplitude and phase coherence of steady-state visual evoked potential (SSVEP) and found that the modulation latency of phase coherence rather than that of amplitude was consistent with the latency of behavioral performance. In this study, as a complementary report, we compared the time course of visual attention shift measured by event-related potentials (ERPs) with that by target detection task. We developed a novel technique to compare ERPs with behavioral results and analyzed the EEG data in our previous study. Two sets of flickering stimulus at different frequencies were presented in the left and right visual hemifields, and a target or distracter pattern was presented randomly at various moments after an attention-cue presentation. The observers were asked to detect targets on the attended stimulus after the cue. We found that two ERP components, P300 and N2pc, were elicited by the target presented at the attended location. Time-course analyses revealed that attentional modulation of the P300 and N2pc amplitudes increased gradually until reaching a maximum and lasted at least 1.5 s after the cue onset, which is similar to the temporal dynamics of behavioral performance. However, attentional modulation of these ERP components started later than that of behavioral performance. Rather, the time course of attentional modulation of behavioral performance was more closely associated with that of the concurrently recorded SSVEPs analyzed. These results suggest that neural activities reflected not by either the P300 or N2pc, but by the SSVEPs, are the source of attentional modulation of behavioral performance.     

Distinct Features of Auditory Steady-State Responses as Compared to Transient Event-Related Potentials

Transient event-related potentials (ERPs) and steady-state responses (SSRs) have been popularly employed to investigate the function of the human brain, but their relationship still remains a matter of debate. Some researchers believed that SSRs could be explained by the linear summation of successive transient ERPs (superposition hypothesis), while others believed that SSRs were the result of the entrainment of a neural rhythm driven by the periodic repetition of a sensory stimulus (oscillatory entrainment hypothesis). In the present study, taking auditory modality as an example, we aimed to clarify the distinct features of SSRs, evoked by the 40-Hz and 60-Hz periodic auditory stimulation, as compared to transient ERPs, evoked by a single click. We observed that (1) SSRs were mainly generated by phase synchronization, while late latency responses (LLRs) in transient ERPs were mainly generated by power enhancement; (2) scalp topographies of LLRs in transient ERPs were markedly different from those of SSRs; (3) the powers of both 40-Hz and 60-Hz SSRs were significantly correlated, while they were not significantly correlated with the N1 power in transient ERPs; (4) whereas SSRs were dominantly modulated by stimulus intensity, middle latency responses (MLRs) were not significantly modulated by both stimulus intensity and subjective loudness judgment, and LLRs were significantly modulated by subjective loudness judgment even within the same stimulus intensity. All these findings indicated that high-frequency SSRs were different from both MLRs and LLRs in transient ERPs, thus supporting the possibility of oscillatory entrainment hypothesis to the generation of SSRs. Therefore, SSRs could be used to explore distinct neural responses as compared to transient ERPs, and help us reveal novel and reliable neural mechanisms of the human brain.     

Unsupervised Feature Learning Improves Prediction of Human Brain Activity in Response to Natural Images

Encoding and decoding in functional magnetic resonance imaging has recently emerged as an area of research to noninvasively characterize the relationship between stimulus features and human brain activity. To overcome the challenge of formalizing what stimulus features should modulate single voxel responses, we introduce a general approach for making directly testable predictions of single voxel responses to statistically adapted representations of ecologically valid stimuli. These representations are learned from unlabeled data without supervision. Our approach is validated using a parsimonious computational model of (i) how early visual cortical representations are adapted to statistical regularities in natural images and (ii) how populations of these representations are pooled by single voxels. This computational model is used to predict single voxel responses to natural images and identify natural images from stimulus-evoked multiple voxel responses. We show that statistically adapted low-level sparse and invariant representations of natural images better span the space of early visual cortical representations and can be more effectively exploited in stimulus identification than hand-designed Gabor wavelets. Our results demonstrate the potential of our approach to better probe unknown cortical representations.     

How the Visual Cortex Handles Stimulus Noise: Insights from Amblyopia

Adding noise to a visual image makes object recognition more effortful and has a widespread effect on human electrophysiological responses. However, visual cortical processes directly involved in handling the stimulus noise have yet to be identified and dissociated from the modulation of the neural responses due to the deteriorated structural information and increased stimulus uncertainty in the case of noisy images. Here we show that the impairment of face gender categorization performance in the case of noisy images in amblyopic patients correlates with amblyopic deficits measured in the noise-induced modulation of the P1/P2 components of single-trial event-related potentials (ERP). On the other hand, the N170 ERP component is similarly affected by the presence of noise in the two eyes and its modulation does not predict the behavioral deficit. These results have revealed that the efficient processing of noisy images depends on the engagement of additional processing resources both at the early, feature-specific as well as later, object-level stages of visual cortical processing reflected in the P1 and P2 ERP components, respectively. Our findings also suggest that noise-induced modulation of the N170 component might reflect diminished face-selective neuronal responses to face images with deteriorated structural information.     

Cortical Surround Interactions and Perceptual Salience via Natural Scene Statistics

Spatial context in images induces perceptual phenomena associated with salience and modulates the responses of neurons in primary visual cortex (V1). However, the computational and ecological principles underlying contextual effects are incompletely understood. We introduce a model of natural images that includes grouping and segmentation of neighboring features based on their joint statistics, and we interpret the firing rates of V1 neurons as performing optimal recognition in this model. We show that this leads to a substantial generalization of divisive normalization, a computation that is ubiquitous in many neural areas and systems. A main novelty in our model is that the influence of the context on a target stimulus is determined by their degree of statistical dependence. We optimized the parameters of the model on natural image patches, and then simulated neural and perceptual responses on stimuli used in classical experiments. The model reproduces some rich and complex response patterns observed in V1, such as the contrast dependence, orientation tuning and spatial asymmetry of surround suppression, while also allowing for surround facilitation under conditions of weak stimulation. It also mimics the perceptual salience produced by simple displays, and leads to readily testable predictions. Our results provide a principled account of orientation-based contextual modulation in early vision and its sensitivity to the homogeneity and spatial arrangement of inputs, and lends statistical support to the theory that V1 computes visual salience.     

Evoked responses induced by random polygonal pattern presentation.

An experiment is described which was designated to test the assumption that visual EEG evoked response waveshapes in human beings may be influenced by the presentation of random shapes differing in level of their similarity and in angular position. A clear-cut difference between the blank flash and the patterns which might be determined by the presence of contours was observed. However, neither differences in EEG evoked responses referred to the shape nor to the angular position of the polygons were found.     

Inhibitory Control in Bilinguals and Musicians: Event Related Potential (ERP) Evidence for Experience-Specific Effects

Bilinguals and musicians exhibit behavioral advantages on tasks with high demands on executive functioning, particularly inhibitory control, but the brain mechanisms supporting these differences are unclear. Of key interest is whether these forms of experience influence cognition through similar or distinct information processing mechanisms. Here, we recorded event-related potentials (ERPs) in three groups – bilinguals, musicians, and controls – who completed a visual go-nogo task that involved the withholding of key presses to rare targets. Participants in each group achieved similar accuracy rates and responses times but the analysis of cortical responses revealed significant differences in ERP waveforms. Success in withholding a prepotent response was associated with enhanced stimulus-locked N2 and P3 wave amplitude relative to go trials. For nogo trials, there were altered timing-specific ERP differences and graded amplitude differences observed in the neural responses across groups. Specifically, musicians showed an enhanced early P2 response accompanied by reduced N2 amplitude whereas bilinguals showed increased N2 amplitude coupled with an increased late positivity wave relative to controls. These findings demonstrate that bilingualism and music training have differential effects on the brain networks supporting executive control over behavior.     

Motor Training Increases the Stability of Activation Patterns in the Primary Motor Cortex

Learning to be skillful is an endowed talent of humans, but neural mechanisms underlying behavioral improvement remain largely unknown. Some studies have reported that the mean magnitude of neural activation is increased after learning, whereas others have instead shown decreased activation. In this study, we used functional magnetic resonance imaging (fMRI) to investigate learning-induced changes in the neural activation in the human brain with a classic motor training task. Specifically, instead of comparing the mean magnitudes of activation before and after training, we analyzed the learning-induced changes in multi-voxel spatial patterns of neural activation. We observed that the stability of the activation patterns, or the similarity of the activation patterns between the even and odd runs of the fMRI scans, was significantly increased in the primary motor cortex (M1) after training. By contrast, the mean magnitude of neural activation remained unchanged. Therefore, our study suggests that learning shapes the brain by increasing the stability of the activation patterns, therefore providing a new perspective in understanding the neural mechanisms underlying learning.     

Semantic Wavelet-Induced Frequency-Tagging (SWIFT) Periodically Activates Category Selective Areas While Steadily Activating Early Visual Areas

Primate visual systems process natural images in a hierarchical manner: at the early stage, neurons are tuned to local image features, while neurons in high-level areas are tuned to abstract object categories. Standard models of visual processing assume that the transition of tuning from image features to object categories emerges gradually along the visual hierarchy. Direct tests of such models remain difficult due to confounding alteration in low-level image properties when contrasting distinct object categories. When such contrast is performed in a classic functional localizer method, the desired activation in high-level visual areas is typically accompanied with activation in early visual areas. Here we used a novel image-modulation method called SWIFT (semantic wavelet-induced frequency-tagging), a variant of frequency-tagging techniques. Natural images modulated by SWIFT reveal object semantics periodically while keeping low-level properties constant. Using functional magnetic resonance imaging (fMRI), we indeed found that faces and scenes modulated with SWIFT periodically activated the prototypical category-selective areas while they elicited sustained and constant responses in early visual areas. SWIFT and the localizer were selective and specific to a similar extent in activating category-selective areas. Only SWIFT progressively activated the visual pathway from low- to high-level areas, consistent with predictions from standard hierarchical models. We confirmed these results with criterion-free methods, generalizing the validity of our approach and show that it is possible to dissociate neural activation in early and category-selective areas. Our results provide direct evidence for the hierarchical nature of the representation of visual objects along the visual stream and open up future applications of frequency-tagging methods in fMRI.     

Differences Help Recognition: A Probabilistic Interpretation

This paper presents a computational model to address one prominent psychological behavior of human beings to recognize images. The basic pursuit of our method can be concluded as that differences among multiple images help visual recognition. Generally speaking, we propose a statistical framework to distinguish what kind of image features capture sufficient category information and what kind of image features are common ones shared in multiple classes. Mathematically, the whole formulation is subject to a generative probabilistic model. Meanwhile, a discriminative functionality is incorporated into the model to interpret the differences among all kinds of images. The whole Bayesian formulation is solved in an Expectation-Maximization paradigm. After finding those discriminative patterns among different images, we design an image categorization algorithm to interpret how these differences help visual recognition within the bag-of-feature framework. The proposed method is verified on a variety of image categorization tasks including outdoor scene images, indoor scene images as well as the airborne SAR images from different perspectives.     

Long-range, pattern-dependent contextual effects in early human visual cortex

The standard view of neurons in early visual cortex is that they behave like localized feature detectors. Here we demonstrate that processing in early visual areas goes beyond feature detection by showing that neural responses are greater when a feature deviates from its context compared to when it does not deviate from its context. Using psychophysics, fMRI, and electroencephalography methodologies, we measured neural responses to an oriented Gabor ("target") embedded in various visual patterns as defined by the relative orientation of flanking stimuli. We first show using psychophysical contrast adaptation and fMRI that a target that differs from its context results in more neural activity compared to a target that is contained within an alternating sequence, suggesting that neurons in early visual cortex are sensitive to large-scale orientation patterns. Next, we use event-related potentials to show that orientation deviations affect the earliest sensory components of the target response. Finally, we use forced-choice classification of "noise" stimuli to show that we are more likely to "see" orientations that deviate from the context. Our results suggest that early visual cortex is sensitive to global patterns in images in a way that is markedly different from the predictions of standard models of cortical visual processing.     

A New Standardized Stimulus Set for Studying Need-of-Help Recognition (NeoHelp)

This article presents the NeoHelp visual stimulus set created to facilitate investigation of need-of-help recognition with clinical and normative populations of different ages, including children. Need-of-help recognition is one aspect of socioemotional development and a necessary precondition for active helping. The NeoHelp consists of picture pairs showing everyday situations: The first item in a pair depicts a child needing help to achieve a goal; the second one shows the child achieving the goal. Pictures of birds in analogue situations are also included. These control stimuli enable implementation of a human-animal categorization task which serves to separate behavioral correlates specific to need-of-help recognition from general differentiation processes. It is a concern in experimental research to ensure that results do not relate to systematic perceptual differences when comparing responses to categories of different content. Therefore, we not only derived the NeoHelp-pictures within a pair from one another by altering as little as possible, but also assessed their perceptual similarity empirically. We show that NeoHelp-picture pairs are very similar regarding low-level perceptual properties across content categories. We obtained data from 60 children in a broad age range (4 to 13 years) for three different paradigms, in order to assess whether the intended categorization and differentiation could be observed reliably in a normative population. Our results demonstrate that children can differentiate the pictures'' content regarding both need-of-help category as well as species as intended in spite of the high perceptual similarities. We provide standard response characteristics (hit rates and response times) that are useful for future selection of stimuli and comparison of results across studies. We show that task requirements coherently determine which aspects of the pictures influence response characteristics. Thus, we present NeoHelp, the first open-access standardized visual stimuli set for investigation of need-of-help recognition and invite researchers to use and extend it.     

An ERP Analysis of Recognition and Categorization Decisions in a Prototype-Distortion Task

Background

Theories of categorization make different predictions about the underlying processes used to represent categories. Episodic theories suggest that categories are represented in memory by storing previously encountered exemplars in memory. Prototype theories suggest that categories are represented in the form of a prototype independently of memory. A number of studies that show dissociations between categorization and recognition are often cited as evidence for the prototype account. These dissociations have compared recognition judgements made to one set of items to categorization judgements to a different set of items making a clear interpretation difficult. Instead of using different stimuli for different tests this experiment compares the processes by which participants make decisions about category membership in a prototype-distortion task and with recognition decisions about the same set of stimuli by examining the Event Related Potentials (ERPs) associated with them.

Method

Sixty-three participants were asked to make categorization or recognition decisions about stimuli that either formed an artificial category or that were category non-members. We examined the ERP components associated with both kinds of decision for pre-exposed and control participants.

Conclusion

In contrast to studies using different items we observed no behavioural differences between the two kinds of decision; participants were equally able to distinguish category members from non-members, regardless of whether they were performing a recognition or categorisation judgement. Interestingly, this did not interact with prior-exposure. However, the ERP data demonstrated that the early visual evoked response that discriminated category members from non-members was modulated by which judgement participants performed and whether they had been pre-exposed to category members. We conclude from this that any differences between categorization and recognition reflect differences in the information that participants focus on in the stimuli to make the judgements at test, rather than any differences in encoding or process.     

Sequence learning recodes cortical representations instead of strengthening initial ones

We contrast two computational models of sequence learning. The associative learner posits that learning proceeds by strengthening existing association weights. Alternatively, recoding posits that learning creates new and more efficient representations of the learned sequences. Importantly, both models propose that humans act as optimal learners but capture different statistics of the stimuli in their internal model. Furthermore, these models make dissociable predictions as to how learning changes the neural representation of sequences. We tested these predictions by using fMRI to extract neural activity patterns from the dorsal visual processing stream during a sequence recall task. We observed that only the recoding account can explain the similarity of neural activity patterns, suggesting that participants recode the learned sequences using chunks. We show that associative learning can theoretically store only very limited number of overlapping sequences, such as common in ecological working memory tasks, and hence an efficient learner should recode initial sequence representations.