Knowledge-based vision and simple visual machines.

The vast majority of work in machine vision emphasizes the representation of perceived objects and events: it is these internal representations that incorporate the ''knowledge'' in knowledge-based vision or form the ''models'' in model-based vision. In this paper, we discuss simple machine vision systems developed by artificial evolution rather than traditional engineering design techniques, and note that the task of identifying internal representations within such systems is made difficult by the lack of an operational definition of representation at the causal mechanistic level. Consequently, we question the nature and indeed the existence of representations posited to be used within natural vision systems (i.e. animals). We conclude that representations argued for on a priori grounds by external observers of a particular vision system may well be illusory, and are at best place-holders for yet-to-be-identified causal mechanistic interactions. That is, applying the knowledge-based vision approach in the understanding of evolved systems (machines or animals) may well lead to theories and models that are internally consistent, computationally plausible, and entirely wrong.     

Are There Multiple Visual Short-Term Memory Stores?

Background

Classic work on visual short-term memory (VSTM) suggests that people store a limited amount of items for subsequent report. However, when human observers are cued to shift attention to one item in VSTM during retention, it seems as if there is a much larger representation, which keeps additional items in a more fragile VSTM store. Thus far, it is not clear whether the capacity of this fragile VSTM store indeed exceeds the traditional capacity limits of VSTM. The current experiments address this issue and explore the capacity, stability, and duration of fragile VSTM representations.

Methodology/Principal Findings

We presented cues in a change-detection task either just after off-set of the memory array (iconic-cue), 1,000 ms after off-set of the memory array (retro-cue) or after on-set of the probe array (post-cue). We observed three stages in visual information processing 1) iconic memory with unlimited capacity, 2) a four seconds lasting fragile VSTM store with a capacity that is at least a factor of two higher than 3) the robust and capacity-limited form of VSTM. Iconic memory seemed to depend on the strength of the positive after-image resulting from the memory display and was virtually absent under conditions of isoluminance or when intervening light masks were presented. This suggests that iconic memory is driven by prolonged retinal activation beyond stimulus duration. Fragile VSTM representations were not affected by light masks, but were completely overwritten by irrelevant pattern masks that spatially overlapped the memory array.

Conclusions/Significance

We find that immediately after a stimulus has disappeared from view, subjects can still access information from iconic memory because they can see an after-image of the display. After that period, human observers can still access a substantial, but somewhat more limited amount of information from a high-capacity, but fragile VSTM that is overwritten when new items are presented to the eyes. What is left after that is the traditional VSTM store, with a limit of about four objects. We conclude that human observers store more sustained representations than is evident from standard change detection tasks and that these representations can be accessed at will.     

Sensory exploitation and cultural transmission: the late emergence of iconic representations in human evolution

Iconic representations (i.e., figurative imagery and realistic art) only started to appear consistently some 45,000 years ago, although humans have been anatomically modern since 200,000–160,000 years ago. What explains this? Some authors have suggested a neurocognitive change took place, leading to a creative explosion, although this has been contested. Here, we examine the hypothesis that demographic changes caused cultural “cumulative adaptive evolution” and as such the emergence of modern symbolic behavior. This approach usefully explains the evolution of utilitarian skills and tools, and the creation of symbols to identify groups. However, it does not equally effectively explain the evolution of behaviors that may not be directly adaptive, such as the production of iconic representations like figurines and rock art. In order to shed light on their emergence, we propose to combine the above-mentioned cultural hypothesis with the concept of sensory exploitation. The concept essentially states that behavioral traits (in this case iconic art production) which exploit pre-existing sensory sensitivities will evolve if not hindered by costs (i.e., natural selection). In this view, iconic art traditions are evolved by piggybacking on cumulative adaptive evolution. Since it is to date uncertain whether art has served any adaptive function in human evolution, parsimony demands paying more attention to the primary and afunctional mechanism of sensory exploitation as opposed to mechanisms of models based exclusively on secondary benefits (such as Miller’s, for instance, in which art is proposed to evolve as a sexual display of fitness).     

3D current-voltage-time surfaces unveil critical repolarization differences underlying similar cardiac action potentials: A model study

The number of mathematical models of cardiac cellular excitability is rapidly growing, and compact graphical representations of their properties can make new acquisitions available for a broader range of scientists in cardiac field. Particularly, the intrinsic over-determination of the model equations systems when fitted only to action potential (AP) waveform and the fact that they are frequently tuned on data covering only a relatively narrow range of dynamic conditions, often lead modellers to compare very similar AP profiles, which underlie though quite different excitable properties. In this study I discuss a novel compact 3D representation of the cardiac cellular AP, where the third dimension represents the instantaneous current–voltage profile of the membrane, measured as repolarization proceeds. Measurements of this type have been used previously for in vivo experiments, and are adopted here iteratively at a very high time, voltage, current-resolution on (i) the same human ventricular model, endowed with two different parameters sets which generate the same AP waveform, and on (ii) three different models of the same human ventricular cell type. In these 3D representations, the AP waveforms lie at the intersection between instantaneous time–voltage–current surfaces and the zero-current plane. Different surfaces can share the same intersection and therefore the same AP; in these cases, the morphology of the current surface provides a compact view of important differences within corresponding repolarization dynamics.Refractory period, supernormal excitability window, and extent of repolarization reserve can be visualized at once. Two pivotal dynamical properties can be precisely assessed, i.e. all-or-nothing repolarization window and membrane resistance during recovery. I discuss differences in these properties among the membranes under study, and show relevant implications for cardiac cellular repolarization.     

Cultural evolution and perpetuation of arbitrary communicative conventions in experimental microsocieties

Previous studies have shown that iconic graphical signs can evolve into symbols through repeated usage within dyads and interacting communities. Here we investigate the evolution of graphical signs over chains of participants. In these chains (or “replacement microsocieties”), membership of an interacting group changed repeatedly such that the most experienced members were continually replaced by naïve participants. Signs rapidly became symbolic, such that they were mutually incomprehensible across experienced members of different chains, and new entrants needed to learn conventionalised meanings. An objective measure of graphical complexity (perimetric complexity) showed that the signs used within the microsocieties were becoming progressively simplified over successive usage. This is the first study to show that the signs that evolve in graphical communication experiments can be transmitted to, and spontaneously adopted by, naïve participants. This provides critical support for the view that human communicative symbols could have evolved culturally from iconic representations.     

Deep Supervised,but Not Unsupervised,Models May Explain IT Cortical Representation

Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.     

Spontaneous Reorientation Is Guided by Perceived Surface Distance,Not by Image Matching Or Comparison

Humans and animals recover their sense of position and orientation using properties of the surface layout, but the processes underlying this ability are disputed. Although behavioral and neurophysiological experiments on animals long have suggested that reorientation depends on representations of surface distance, recent experiments on young children join experimental studies and computational models of animal navigation to suggest that reorientation depends either on processing of any continuous perceptual variables or on matching of 2D, depthless images of the landscape. We tested the surface distance hypothesis against these alternatives through studies of children, using environments whose 3D shape and 2D image properties were arranged to enhance or cancel impressions of depth. In the absence of training, children reoriented by subtle differences in perceived surface distance under conditions that challenge current models of 2D-image matching or comparison processes. We provide evidence that children’s spontaneous navigation depends on representations of 3D layout geometry.     

The Importance of Subcellular Structures to the Modeling of Biological Cells in the Context of Computational Bioelectromagnetics Simulations

Numerical investigation of the interaction of electromagnetic fields with eukaryotic cells requires specifically adapted computer models. Virtual microdosimetry, used to investigate exposure, requires volumetric cell models, which are numerically challenging. For this reason, a method is presented here to determine the current and volumetric loss densities occurring in single cells and their distinct compartments in a spatially accurate manner as a first step toward multicellular models within the microstructure of tissue layers. To achieve this, 3D models of the electromagnetic exposure of generic eukaryotic cells of different shape (i.e. spherical and ellipsoidal) and internal complexity (i.e. different organelles) are performed in a virtual, finite element method-based capacitor experiment in the frequency range from 10 Hz to 100 GHz. In this context, the spectral response of the current and loss distribution within the cell compartments is investigated and any effects that occur are attributed either to the dispersive material properties of these compartments or to the geometric characteristics of the cell model investigated in each case. In these investigations, the cell is represented as an anisotropic body with an internal distributed membrane system of low conductivity that mimics the endoplasmic reticulum in a simplified manner. This will be used to determine which details of the cell interior need to be modeled, how the electric field and the current density will be distributed in this region, and where the electromagnetic energy is absorbed in the microstructure regarding electromagnetic microdosimetry. Results show that for 5 G frequencies, membranes make a significant contribution to the absorption losses. © 2023 The Authors. Bioelectromagnetics published by Wiley Periodicals LLC on behalf of Bioelectromagnetics Society.     

Thermodynamic characterization of RNA 2 × 3 nucleotide internal loops

To better elucidate RNA structure-function relationships and to improve the design of pharmaceutical agents that target specific RNA motifs, an understanding of RNA primary, secondary, and tertiary structure is necessary. The prediction of RNA secondary structure from sequence is an intermediate step in predicting RNA three-dimensional structure. RNA secondary structure is typically predicted using a nearest neighbor model based on free energy parameters. The current free energy parameters for 2 × 3 nucleotide loops are based on a 23-member data set of 2 × 3 loops and internal loops of other sizes. A database of representative RNA secondary structures was searched to identify 2 × 3 nucleotide loops that occur in nature. Seventeen of the most frequent 2 × 3 nucleotide loops in this database were studied by optical melting experiments. Fifteen of these loops melted in a two-state manner, and the associated experimental ΔG°(37,2×3) values are, on average, 0.6 and 0.7 kcal/mol different from the values predicted for these internal loops using the predictive models proposed by Lu, Turner, and Mathews [Lu, Z. J., Turner, D. H., and Mathews, D. H. (2006) Nucleic Acids Res. 34, 4912-4924] and Chen and Turner [Chen, G., and Turner, D. H. (2006) Biochemistry 45, 4025-4043], respectively. These new ΔG°(37,2×3) values can be used to update the current algorithms that predict secondary structure from sequence. To improve free energy calculations for duplexes containing 2 × 3 nucleotide loops that still do not have experimentally determined free energy contributions, an updated predictive model was derived. This new model resulted from a linear regression analysis of the data reported here combined with 31 previously studied 2 × 3 nucleotide internal loops. Most of the values for the parameters in this new predictive model are within experimental error of those of the previous models, suggesting that approximations and assumptions associated with the derivation of the previous nearest neighbor parameters were valid. The updated predictive model predicts free energies of 2 × 3 nucleotide internal loops within 0.4 kcal/mol, on average, of the experimental free energy values. Both the experimental values and the updated predictive model can be used to improve secondary structure prediction from sequence.     

Size-Sensitive Perceptual Representations Underlie Visual and Haptic Object Recognition

A variety of similarities between visual and haptic object recognition suggests that the two modalities may share common representations. However, it is unclear whether such common representations preserve low-level perceptual features or whether transfer between vision and haptics is mediated by high-level, abstract representations. Two experiments used a sequential shape-matching task to examine the effects of size changes on unimodal and crossmodal visual and haptic object recognition. Participants felt or saw 3D plastic models of familiar objects. The two objects presented on a trial were either the same size or different sizes and were the same shape or different but similar shapes. Participants were told to ignore size changes and to match on shape alone. In Experiment 1, size changes on same-shape trials impaired performance similarly for both visual-to-visual and haptic-to-haptic shape matching. In Experiment 2, size changes impaired performance on both visual-to-haptic and haptic-to-visual shape matching and there was no interaction between the cost of size changes and direction of transfer. Together the unimodal and crossmodal matching results suggest that the same, size-specific perceptual representations underlie both visual and haptic object recognition, and indicate that crossmodal memory for objects must be at least partly based on common perceptual representations.     

Internal representations for associative memory

We describe a class of feed forward neural network models for associative content addressable memory (ACAM) which utilize sparse internal representations for stored data. In addition to the input and output layers, our networks incorporate an intermediate processing layer which serves to label each stored memory and to perform error correction and association. We study two classes of internal label representations: the unary representation and various sparse, distributed representations. Finally, we consider storage of sparse data and sparsification of data. These models are found to have advantages in terms of storage capacity, hardware efficiency, and recall reliability when compared to the Hopfield model, and to possess analogies to both biological neural networks and standard digital computer memories.     

Measuring internal representations from behavioral and brain data

The study of internal knowledge representations is a cornerstone of the research agenda in the interdisciplinary study of cognition. An influential proposal assumes that the brain uses its internal knowledge of the external world to constrain, in a top-down manner, high-dimensional sensory data into a lower-dimensional representation that enables perceptual decisions and other higher-level cognitive functions [1-9]. This proposal relies on a precise formulation of the observer-specific internal knowledge (i.e., the internal representations, or models) that guides reduction of the high-dimensional retinal input onto a low-dimensional code. Here, we directly revealed the content of subjective internal representations by instructing five observers to detect a face in the presence of only white noise, to force a pure top-down, knowledge-based task. We used reverse correlation methods to visualize each observer's internal representation that supports detection of an illusory face. Using reverse correlation again, this time applied to observers' electroencephalogram activity, we established where and when in the brain specific internal knowledge conceptually interprets the input white noise as a face. We show that internal representations can be reconstructed experimentally from behavioral and brain data, and that their content drives neural activity first over frontal and then over occipitotemporal cortex.     

How Lateral Connections and Spiking Dynamics May Separate Multiple Objects Moving Together

Over successive stages, the ventral visual system of the primate brain develops neurons that respond selectively to particular objects or faces with translation, size and view invariance. The powerful neural representations found in Inferotemporal cortex form a remarkably rapid and robust basis for object recognition which belies the difficulties faced by the system when learning in natural visual environments. A central issue in understanding the process of biological object recognition is how these neurons learn to form separate representations of objects from complex visual scenes composed of multiple objects. We show how a one-layer competitive network comprised of ‘spiking’ neurons is able to learn separate transformation-invariant representations (exemplified by one-dimensional translations) of visual objects that are always seen together moving in lock-step, but separated in space. This is achieved by combining ‘Mexican hat’ functional lateral connectivity with cell firing-rate adaptation to temporally segment input representations of competing stimuli through anti-phase oscillations (perceptual cycles). These spiking dynamics are quickly and reliably generated, enabling selective modification of the feed-forward connections to neurons in the next layer through Spike-Time-Dependent Plasticity (STDP), resulting in separate translation-invariant representations of each stimulus. Variations in key properties of the model are investigated with respect to the network’s ability to develop appropriate input representations and subsequently output representations through STDP. Contrary to earlier rate-coded models of this learning process, this work shows how spiking neural networks may learn about more than one stimulus together without suffering from the ‘superposition catastrophe’. We take these results to suggest that spiking dynamics are key to understanding biological visual object recognition.     

Hearing and seeing meaning in speech and gesture: insights from brain and behaviour

As we speak, we use not only the arbitrary form–meaning mappings of the speech channel but also motivated form–meaning correspondences, i.e. iconic gestures that accompany speech (e.g. inverted V-shaped hand wiggling across gesture space to demonstrate walking). This article reviews what we know about processing of semantic information from speech and iconic gestures in spoken languages during comprehension of such composite utterances. Several studies have shown that comprehension of iconic gestures involves brain activations known to be involved in semantic processing of speech: i.e. modulation of the electrophysiological recording component N400, which is sensitive to the ease of semantic integration of a word to previous context, and recruitment of the left-lateralized frontal–posterior temporal network (left inferior frontal gyrus (IFG), medial temporal gyrus (MTG) and superior temporal gyrus/sulcus (STG/S)). Furthermore, we integrate the information coming from both channels recruiting brain areas such as left IFG, posterior superior temporal sulcus (STS)/MTG and even motor cortex. Finally, this integration is flexible: the temporal synchrony between the iconic gesture and the speech segment, as well as the perceived communicative intent of the speaker, modulate the integration process. Whether these findings are special to gestures or are shared with actions or other visual accompaniments to speech (e.g. lips) or other visual symbols such as pictures are discussed, as well as the implications for a multimodal view of language.     

The sound symbolism bootstrapping hypothesis for language acquisition and language evolution

Sound symbolism is a non-arbitrary relationship between speech sounds and meaning. We review evidence that, contrary to the traditional view in linguistics, sound symbolism is an important design feature of language, which affects online processing of language, and most importantly, language acquisition. We propose the sound symbolism bootstrapping hypothesis, claiming that (i) pre-verbal infants are sensitive to sound symbolism, due to a biologically endowed ability to map and integrate multi-modal input, (ii) sound symbolism helps infants gain referential insight for speech sounds, (iii) sound symbolism helps infants and toddlers associate speech sounds with their referents to establish a lexical representation and (iv) sound symbolism helps toddlers learn words by allowing them to focus on referents embedded in a complex scene, alleviating Quine''s problem. We further explore the possibility that sound symbolism is deeply related to language evolution, drawing the parallel between historical development of language across generations and ontogenetic development within individuals. Finally, we suggest that sound symbolism bootstrapping is a part of a more general phenomenon of bootstrapping by means of iconic representations, drawing on similarities and close behavioural links between sound symbolism and speech-accompanying iconic gesture.