Towards a general theory of neural computation based on prediction by single neurons

Although there has been tremendous progress in understanding the mechanics of the nervous system, there has not been a general theory of its computational function. Here I present a theory that relates the established biophysical properties of single generic neurons to principles of Bayesian probability theory, reinforcement learning and efficient coding. I suggest that this theory addresses the general computational problem facing the nervous system. Each neuron is proposed to mirror the function of the whole system in learning to predict aspects of the world related to future reward. According to the model, a typical neuron receives current information about the state of the world from a subset of its excitatory synaptic inputs, and prior information from its other inputs. Prior information would be contributed by synaptic inputs representing distinct regions of space, and by different types of non-synaptic, voltage-regulated channels representing distinct periods of the past. The neuron's membrane voltage is proposed to signal the difference between current and prior information ("prediction error" or "surprise"). A neuron would apply a Hebbian plasticity rule to select those excitatory inputs that are the most closely correlated with reward but are the least predictable, since unpredictable inputs provide the neuron with the most "new" information about future reward. To minimize the error in its predictions and to respond only when excitation is "new and surprising," the neuron selects amongst its prior information sources through an anti-Hebbian rule. The unique inputs of a mature neuron would therefore result from learning about spatial and temporal patterns in its local environment, and by extension, the external world. Thus the theory describes how the structure of the mature nervous system could reflect the structure of the external world, and how the complexity and intelligence of the system might develop from a population of undifferentiated neurons, each implementing similar learning algorithms.     

Geographic variation in hybridization across a reinforcement contact zone of chorus frogs (Pseudacris)

Reinforcement contact zones, which are secondary contact zones where species are diverging in reproductive behaviors due to selection against hybridization, represent natural laboratories for studying speciation‐in‐action. Here, we examined replicate localities across the entire reinforcement contact zone between North American chorus frogs Pseudacris feriarum and P. nigrita to investigate geographic variation in hybridization frequencies and to assess whether reinforcement may have contributed to increased genetic divergence within species. Previous work indicated these species have undergone reproductive character displacement (RCD) in male acoustic signals and female preferences due to reinforcement. We also examined acoustic signal variation across the contact zone to assess whether signal characteristics reliably predict hybrid index and to elucidate whether the degree of RCD predicts hybridization rate. Using microsatellites, mitochondrial sequences, and acoustic signal information from >1,000 individuals across >50 localities and ten sympatric focal regions, we demonstrate: (1) hybridization occurs and (2) varies substantially across the geographic range of the contact zone, (3) hybridization is asymmetric and in the direction predicted from observed patterns of asymmetric RCD, (4) in one species, genetic distance is higher between conspecific localities where one or both have been reinforced than between nonreinforced localities, after controlling for geographic distance, (5) acoustic signal characters strongly predict hybrid index, and (6) the degree of RCD does not strongly predict admixture levels. By showing that hybridization occurs in all sympatric localities, this study provides the fifth and final line of evidence that reproductive character displacement is due to reinforcement in the chorus frog contact zone. Furthermore, this work suggests that the dual action of cascade reinforcement and partial geographic isolation is promoting genetic diversification within one of the reinforced species.     

Explanations in search of observations

The paper explores how, in economics and biology, theoretical models are used as explanatory devices. It focuses on a modelling strategy by which, instead of starting with an unexplained regularity in the world, the modeller begins by creating a credible model world. The model world exhibits a regularity, induced by a mechanism in that world. The modeller concludes that there may be a part of the real world in which a similar regularity occurs and that, were that the case, the model would offer an explanation. Little concrete guidance is given about where such a regularity might be found. Three modelling exercises in evolutionary game theory—one from economics and two from biology—are used as case studies. Two of these (one from each discipline) exemplify ‘explanation in search of observation’. The third goes a step further, analysing a regularity in a model world and treating it as informative about the real world, but without saying anything about real phenomena. The paper argues that if the relation between the model and real worlds is understood in terms of similarity, and if modelling is understood as an ongoing discovery process rather than as the demonstration of empirical truths, there can be value in creating explanations before finding the regularities that are to be explained.     

Functional Imaging with Reinforcement, Eyetracking, and Physiological Monitoring

We use functional brain imaging (fMRI) to study neural circuits that underlie decision-making. To understand how outcomes affect decision processes, simple perceptual tasks are combined with appetitive and aversive reinforcement. However, the use of reinforcers such as juice and airpuffs can create challenges for fMRI. Reinforcer delivery can cause head movement, which creates artifacts in the fMRI signal. Reinforcement can also lead to changes in heart rate and respiration that are mediated by autonomic pathways. Changes in heart rate and respiration can directly affect the fMRI (BOLD) signal in the brain and can be confounded with signal changes that are due to neural activity. In this presentation, we demonstrate methods for administering reinforcers in a controlled manner, for stabilizing the head, and for measuring pulse and respiration.Open in a separate windowClick here to view.^{(55M, flv)}     

A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems

Error-driven learning rules have received considerable attention because of their close relationships to both optimal theory and neurobiological mechanisms. However, basic forms of these rules are effective under only a restricted set of conditions in which the environment is stable. Recent studies have defined optimal solutions to learning problems in more general, potentially unstable, environments, but the relevance of these complex mathematical solutions to how the brain solves these problems remains unclear. Here, we show that one such Bayesian solution can be approximated by a computationally straightforward mixture of simple error-driven ‘Delta’ rules. This simpler model can make effective inferences in a dynamic environment and matches human performance on a predictive-inference task using a mixture of a small number of Delta rules. This model represents an important conceptual advance in our understanding of how the brain can use relatively simple computations to make nearly optimal inferences in a dynamic world.     

Quantum physics in neuroscience and psychology: a neurophysical model of mind-brain interaction

Neuropsychological research on the neural basis of behaviour generally posits that brain mechanisms will ultimately suffice to explain all psychologically described phenomena. This assumption stems from the idea that the brain is made up entirely of material particles and fields, and that all causal mechanisms relevant to neuroscience can therefore be formulated solely in terms of properties of these elements. Thus, terms having intrinsic mentalistic and/or experiential content (e.g. 'feeling', 'knowing' and 'effort') are not included as primary causal factors. This theoretical restriction is motivated primarily by ideas about the natural world that have been known to be fundamentally incorrect for more than three-quarters of a century. Contemporary basic physical theory differs profoundly from classic physics on the important matter of how the consciousness of human agents enters into the structure of empirical phenomena. The new principles contradict the older idea that local mechanical processes alone can account for the structure of all observed empirical data. Contemporary physical theory brings directly and irreducibly into the overall causal structure certain psychologically described choices made by human agents about how they will act. This key development in basic physical theory is applicable to neuroscience, and it provides neuroscientists and psychologists with an alternative conceptual framework for describing neural processes. Indeed, owing to certain structural features of ion channels critical to synaptic function, contemporary physical theory must in principle be used when analysing human brain dynamics. The new framework, unlike its classic-physics-based predecessor, is erected directly upon, and is compatible with, the prevailing principles of physics. It is able to represent more adequately than classic concepts the neuroplastic mechanisms relevant to the growing number of empirical studies of the capacity of directed attention and mental effort to systematically alter brain function.     

MPR

Mathematical Principles of Reinforcement (MPR) is a theory of reinforcement schedules. This paper reviews the origin of the principles constituting MPR: arousal, association and constraint. Incentives invigorate responses, in particular those preceding and predicting the incentive. The process that generates an associative bond between stimuli, responses and incentives is called coupling. The combination of arousal and coupling constitutes reinforcement. Models of coupling play a central role in the evolution of the theory. The time required to respond constrains the maximum response rates, and generates a hyperbolic relation between rate of responding and rate of reinforcement. Models of control by ratio schedules are developed to illustrate the interaction of the principles. Correlations among parameters are incorporated into the structure of the models, and assumptions that were made in the original theory are refined in light of current data.     

Biophysical models of intrinsic homeostasis: Firing rates and beyond

In view of ever-changing conditions both in the external world and in intrinsic brain states, maintaining the robustness of computations poses a challenge, adequate solutions to which we are only beginning to understand. At the level of cell-intrinsic properties, biophysical models of neurons permit one to identify relevant physiological substrates that can serve as regulators of neuronal excitability and to test how feedback loops can stabilize crucial variables such as long-term calcium levels and firing rates. Mathematical theory has also revealed a rich set of complementary computational properties arising from distinct cellular dynamics and even shaping processing at the network level. Here, we provide an overview over recently explored homeostatic mechanisms derived from biophysical models and hypothesize how multiple dynamical characteristics of cells, including their intrinsic neuronal excitability classes, can be stably controlled.     

Effect of LPS treatment on tyrosine hydroxylase expression and Parkinson-like behaviors

Puberty is a critical period of development during which the brain undergoes reorganizing and remodeling. Exposure to stress during this period is thought to interfere with normal brain development and increase susceptibility to mental illnesses. In female mice, pubertal exposure to lipopolysaccharide (LPS), a bacterial endotoxin, has been shown to alter sexual, anxiety-like, and depression-like behaviors and cognition in an enduring manner. However, the mechanisms underlying these effects remain unknown. The present study examined age and sex difference in tyrosine hydroxylase (TH) expression and dopamine-dependent and Parkinson-like behaviors following LPS treatment. The results show that LPS treatment during adulthood causes an enduring increase in TH expression in many of the brain regions examined. In contrast, there is no change in TH expression following LPS treatment during puberty. However, pubertal LPS treatment induces enduring behavioral deficits in tests of Parkinson-like behaviors, more so in male than in female mice. These results suggest that the low levels of TH following exposure to pubertal immune challenge may predispose mice to Parkinson-like behavior. These findings add to our understanding of stress and immune responses during puberty and their impact on mental health later in life.     

How (and why) the visual control of action differs from visual perception

Vision not only provides us with detailed knowledge of the world beyond our bodies, but it also guides our actions with respect to objects and events in that world. The computations required for vision-for-perception are quite different from those required for vision-for-action. The former uses relational metrics and scene-based frames of reference while the latter uses absolute metrics and effector-based frames of reference. These competing demands on vision have shaped the organization of the visual pathways in the primate brain, particularly within the visual areas of the cerebral cortex. The ventral ‘perceptual’ stream, projecting from early visual areas to inferior temporal cortex, helps to construct the rich and detailed visual representations of the world that allow us to identify objects and events, attach meaning and significance to them and establish their causal relations. By contrast, the dorsal ‘action’ stream, projecting from early visual areas to the posterior parietal cortex, plays a critical role in the real-time control of action, transforming information about the location and disposition of goal objects into the coordinate frames of the effectors being used to perform the action. The idea of two visual systems in a single brain might seem initially counterintuitive. Our visual experience of the world is so compelling that it is hard to believe that some other quite independent visual signal—one that we are unaware of—is guiding our movements. But evidence from a broad range of studies from neuropsychology to neuroimaging has shown that the visual signals that give us our experience of objects and events in the world are not the same ones that control our actions.     

Formal learning theory dissociates brain regions with different temporal integration

Learning can be characterized as the extraction of reliable predictions about stimulus occurrences from past experience. In two experiments, we investigated the interval of temporal integration of previous learning trials in different brain regions using implicit and explicit Pavlovian fear conditioning with a dynamically changing reinforcement regime in an experimental setting. With formal learning theory (the Rescorla-Wagner model), temporal integration is characterized by the learning rate. Using fMRI and this theoretical framework, we are able to distinguish between learning-related brain regions that show long temporal integration (e.g., amygdala) and higher perceptual regions that integrate only over a short period of time (e.g., fusiform face area, parahippocampal place area). This approach allows for the investigation of learning-related changes in brain activation, as it can dissociate brain areas that differ with respect to their integration of past learning experiences by either computing long-term outcome predictions or instantaneous reinforcement expectancies.     

Stimulus-dependent adjustment of reward prediction error in the midbrain

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.     

Learning with incomplete information and the mathematical structure behind it

We investigate the problem of learning with incomplete information as exemplified by learning with delayed reinforcement. We study a two phase learning scenario in which a phase of Hebbian associative learning based on momentary internal representations is supplemented by an ‘unlearning’ phase depending on a graded reinforcement signal. The reinforcement signal quantifies the success-rate globally for a number of learning steps in phase one, and ‘unlearning’ is indiscriminate with respect to associations learnt in that phase. Learning according to this model is studied via simulations and analytically within a student–teacher scenario for both single layer networks and, for a committee machine. Success and speed of learning depend on the ratio λ of the learning rates used for the associative Hebbian learning phase and for the unlearning-correction in response to the reinforcement signal, respectively. Asymptotically perfect generalization is possible only, if this ratio exceeds a critical value λ _c , in which case the generalization error exhibits a power law decay with the number of examples seen by the student, with an exponent that depends in a non-universal manner on the parameter λ. We find these features to be robust against a wide spectrum of modifications of microscopic modelling details. Two illustrative applications—one of a robot learning to navigate a field containing obstacles, and the problem of identifying a specific component in a collection of stimuli—are also provided.     

Apparent position governs contour-element binding by the visual system

An assumption inherent in many models of visual space is that the spatial coordinates of retinal cells implicitly give rise to the perceptual code for position. The results of the experiments reported here, in which it is shown that retinally non-veridical locations of contour elements are used by the visual system for contour-element binding, lend support to a different view. The visual system does not implicitly code position with reference to the labelled locations of retinal cells, but dynamically extracts spatial position from the aggregate result of local computations. These computations may include local spatial relationships between retinal cells, but are not confined to them; other computations, including position derived from local velocity cues, are combined to code the position of objects in the visual world.     

Modeling changes in probabilistic reinforcement learning during adolescence

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants’ performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.     

Objectivity and Illusion in Evolutionary Ethics: Comments on Waller

In this paper I argue that any adequate evolutionary ethical theory needs to account for moral belief as well as for dispositions to behave altruistically. It also needs to be clear whether it is offering us an account of the motivating reasons behind human behaviour or whether it is giving justifying reasons for a particular set of behaviours or, if both, to distinguish them clearly. I also argue that, unless there are some objective moral truths, the evolutionary ethicist cannot offer justifying reasons for a set of behaviours. I use these points to refute Waller's claims that the illusion of objectivity plays a dispensable role in Ruse's theory, that my critique of Ruse's Darwinian metaethics is built on a false dilemma, that there is nothing to be distressed about if morality is not objective, and that ethical beliefs are subject to a kind of causal explanation that undermines their objectivity in a way that scientific beliefs are not.     

The problem of reinforcement. From behavioural control to neurochemical background and the development of psychopathologies

At the first part of the paper a number of conceptions about psychophysiological mechanisms of reinforcement and its role in brain system activity is presented. At the second part, a significance of the mesolimbic and mesocortical dopaminergic systems in reinforcement process and the relationships between different components of behavioural control (signal, memory, actions, etc.) and intensity of dopamine transmission at the level of n. accumbens and frontal cortex are considered. At the final part of the paper the results of development of some psychopathologies and marginal conditions, related to a low or high content of dopamine in the above-mentioned structures, and possible neurophysiological mechanisms of their formation are presented.     

Concepts of rational taxonomy

The problems are discussed related to development of concepts of rational taxonomy and rational classifications (taxonomic systems) in biology. Rational taxonomy is based on the assumption that the key characteristic of rationality is deductive inference of certain partial judgments about reality under study from other judgments taken as more general and a priory true. Respectively, two forms of rationality are discriminated--ontological and epistemological ones. The former implies inference of classifications properties from general (essential) properties of the reality being investigated. The latter implies inference of the partial rules of judgments about classifications from more general (formal) rules. The following principal concepts of ontologically rational biological taxonomy are considered: "crystallographic" approach, inference of the orderliness of organismal diversity from general laws of Nature, inference of the above orderliness from the orderliness of ontogenetic development programs, based on the concept of natural kind and Cassirer's series theory, based on the systemic concept, based on the idea of periodic systems. Various concepts of ontologically rational taxonomy can be generalized by an idea of the causal taxonomy, according to which any biologically sound classification is founded on a contentwise model of biological diversity that includes explicit indication of general causes responsible for that diversity. It is asserted that each category of general causation and respective background model may serve as a basis for a particular ontologically rational taxonomy as a distinctive research program. Concepts of epistemologically rational taxonomy and classifications (taxonomic systems) can be interpreted in terms of application of certain epistemological criteria of substantiation of scientific status of taxonomy in general and of taxonomic systems in particular. These concepts include: consideration of taxonomy consistency from the standpoint of inductive and hypothetico-deductive argumentation schemes and such fundamental criteria of classifications naturalness as their prognostic capabilities; foundation of a theory of "general taxonomy" as a "general logic", including elements of the axiomatic method. The latter concept constitutes a core of the program of general classiology; it is inconsistent due to absence of anything like "general logic". It is asserted that elaboration of a theory of taxonomy as a biological discipline based on the formal principles of epistemological rationality is not feasible. Instead, it is to be elaborated as ontologically rational one based on biologically sound metatheories about biological diversity causes.