The role of the striatum in adaptation learning: a computational model

To investigate the functional role of the striatum in visuo-motor adaptation, we extend the DIRECT-model for visuo-motor reaching movements formulated by Bullock et al.(J Cogn Neurosci 5:408–435,1993) through two parallel loops, each modeling a distinct contribution of the cortico–cerebellar–thalamo–cortical and the cortico–striato–thalamo–cortical networks to visuo-motor adaptation. Based on evidence of Robertson and Miall(Neuroreport 10(5): 1029–1034, 1999), we implement the function of the cortico–cerebellar–thalamo–cortical loop as a module that gradually adapts to small changes in sensorimotor relationships. The cortico–striato–thalamo–cortical loop on the other hand is hypothesized to act as an adaptive search element, guessing new sensorimotor-transformations and reinforcing successful guesses while punishing unsuccessful ones. In a first step, we show that the model reproduces trajectories and error curves of healthy subjects in a two dimensional center-out reaching task with rotated screen cursor visual feedback. In a second step, we disable learning processes in the cortico–striato– thalamo–cortical loop to simulate subjects with Parkinson’s disease (PD), and show that this leads to error curves typical of subjects with PD. We conclude that the results support our hypothesis, i.e., that the role of the cortico–striato–thalamo–cortical loop in visuo-motor adaptation is that of an adaptive search element.     

Role of CAMKII in reinforcement learning: a computational model of glutamate and dopamine signaling pathways

Timely release of dopamine (DA) at the striatum seems to be important for reinforcement learning (RL) mediated by the basal ganglia. Houk et al. (in: Houk et al (eds) Models of information processing in the basal ganglia, (1995) proposed a cellular signaling pathway model to characterize the interaction between DA and glutamate pathways that have a role in RL. The model simulation results, using GENESIS KINETIKIT simulator, point out that there is not only prolongation of duration as proposed by Houk et al. (1995), but also an enhancement in the amplitude of autophosphorylation of CaMKII. Further, the autophosphorylated form of CaMKII may form a basis for the “eligibility trace” condition required in RL. This simulation study is the first of its kind to support the comprehensive theoretical proposal of Houk et al. (1995).     

Efficient reinforcement learning: computational theories, neuroscience and robotics

Reinforcement learning algorithms have provided some of the most influential computational theories for behavioral learning that depends on reward and penalty. After briefly reviewing supporting experimental data, this paper tackles three difficult theoretical issues that remain to be explored. First, plain reinforcement learning is much too slow to be considered a plausible brain model. Second, although the temporal-difference error has an important role both in theory and in experiments, how to compute it remains an enigma. Third, function of all brain areas, including the cerebral cortex, cerebellum, brainstem and basal ganglia, seems to necessitate a new computational framework. Computational studies that emphasize meta-parameters, hierarchy, modularity and supervised learning to resolve these issues are reviewed here, together with the related experimental data.     

Multiple spike initiation zones in a neuron implicated in learning in the leech: a computational model

Sensitization of the defensive shortening reflex in the leech has been linked to a segmentally repeated tri-synaptic positive feedback loop. Serotonin from the R-cell enhances S-cell excitability, S-cell impulses cross an electrical synapse into the C-interneuron, and the C-interneuron excites the R-cell via a glutamatergic synapse. The C-interneuron has two unusual characteristics. First, impulses take longer to propagate from the S soma to the C soma than in the reverse direction. Second, impulses recorded from the electrically unexcitable C soma vary in amplitude when extracellular divalent cation concentrations are elevated, with smaller impulses failing to induce synaptic potentials in the R-cell. A compartmental, computational model was developed to test the sufficiency of multiple, independent spike initiation zones in the C-interneuron to explain these observations. The model displays asymmetric delays in impulse propagation across the S–C electrical synapse and graded impulse amplitudes in the C-interneuron in simulated high divalent cation concentrations.     

Energy and phase orientation mechanisms: a computational model

A computational model is proposed for spatial orientation processing beyond the initial stage of linear filtering in visual cortex. The model accounts for orientation pop-out, edge location and orientation, and bar location and orientation. It naturally extends to higher order orientation symmetries. The model is consistent with much of the current understanding of early processing in mammalian visual cortex. It builds on the notions of orientation and spatial frequency specific simple cells, any subsequent non-linearity, and orientation 'pooling'. The processing treats simple cell energy, real, and imaginary responses in a unified way to generate 'feature maps'. The 'pooling' operation in each case is a discrete Fourier transform of the simple cell responses over orientation. The suggested processing has implications for psychophysics (e.g. providing an explanation of why orientation discrimination thresholds are more than an order of magnitude less than simple cell orientation bandwidths), provides some understanding of the variety of 'complex-cell' properties found in visual cortex, and provides a plausible starting point for subsequent processing.     

Academic learning,a variational model

By assigning coordinates to the information space comprising all knowledge, rigorous mathematical interpretations can be placed on such terms as academic ability, memory and creativity such that these psychometric concepts can be incorporated into a framework of functional analysis which then permits the optimization of long-term academic learning processes through the location of the teaching trajectories in information space which will maximize the knowledge accumulated in a generalized educational system composed of a complex of subject-pupil-teacher interactions. The concepts of discrete and continuous information spaces are discussed in connection with subject-subject, subjectpupil and pupil-pupil interactions, and the advantages of using variational versus dynamic programming methods of optimizing alternative educational systems are evaluated.     

Vector field approximation: a computational paradigm for motor control and learning

Recent experiments in the spinalized frog (Bizzi et al. 1991) have shown that focal microstimulation of a site in the premotor layers in the lumbar grey matter of the spinal cord results in a field of forces acting on the frog's ankle and converging to a single equilibrium position. These experiments suggested that the neural circuits in the spinal cord are organized in a set of control modules that store a few limb postures in the form of convergent force fields acting on the limb's end-point. Here, we investigate how such postural modules can be combined by the central nervous system for generating and representing a wider repertoire of control patterns. Our work is related to some recent investigations by Poggio and Girosi (1990a, b) who have proposed to represent the task of learning scalar maps as a problem of surface approximation. Consistent both with this view and with our experimental findings in the spinal frog, we regard the issue of generating motor repertoires as a problem of vector-field approximation. To this end, we characterize the output of a control module as a basis field (Mussa-Ivaldi 1992), that is as the vectorial equivalent of a basis function. Our theoretical findings indicate that by combining basis fields, the central nervous system may achieve a number of goals such as (1) the generation of a wide repertoire of control patterns and (2) the representation of these control patterns with a set of coefficients that are invariant under coordinate transformations.     

Maturation of the immune response: a computational model

Experimental studies of the effect on antibody affinity of antigen dose and time after immunization show that average affinity increases progressively with time after immunization, and that this increase is greater at lower doses of antigen. In this paper we describe a polyclonal computer model of the immune system that yields all the essential phenomena of affinity maturation, including dose-dependency. Our main findings are (1) the dose-dependency relationship is not produced when typical assumptions regarding B-cell populations and binding reactions are employed, and (2) it is possible to reproduce this dependency by assuming two classes of lymphocytes: generalists and specialists. Generalists have a low threshold for response and produce antibody of low effectiveness, whereas specialists have a high threshold for response, and produce highly effective antibody. We make an analogy between the generalists and a pioneer species in ecological succession, and suggest how the generalists may contribute to a more effective defense against real infections.     

Impaired associative learning in schizophrenia: behavioral and computational studies

Associative learning is a central building block of human cognition and in large part depends on mechanisms of synaptic plasticity, memory capacity and fronto–hippocampal interactions. A disorder like schizophrenia is thought to be characterized by altered plasticity, and impaired frontal and hippocampal function. Understanding the expression of this dysfunction through appropriate experimental studies, and understanding the processes that may give rise to impaired behavior through biologically plausible computational models will help clarify the nature of these deficits. We present a preliminary computational model designed to capture learning dynamics in healthy control and schizophrenia subjects. Experimental data was collected on a spatial-object paired-associate learning task. The task evinces classic patterns of negatively accelerated learning in both healthy control subjects and patients, with patients demonstrating lower rates of learning than controls. Our rudimentary computational model of the task was based on biologically plausible assumptions, including the separation of dorsal/spatial and ventral/object visual streams, implementation of rules of learning, the explicit parameterization of learning rates (a plausible surrogate for synaptic plasticity), and learning capacity (a plausible surrogate for memory capacity). Reductions in learning dynamics in schizophrenia were well-modeled by reductions in learning rate and learning capacity. The synergy between experimental research and a detailed computational model of performance provides a framework within which to infer plausible biological bases of impaired learning dynamics in schizophrenia.     

The computational neurobiology of learning and reward

Following the suggestion that midbrain dopaminergic neurons encode a signal, known as a 'reward prediction error', used by artificial intelligence algorithms for learning to choose advantageous actions, the study of the neural substrates for reward-based learning has been strongly influenced by computational theories. In recent work, such theories have been increasingly integrated into experimental design and analysis. Such hybrid approaches have offered detailed new insights into the function of a number of brain areas, especially the cortex and basal ganglia. In part this is because these approaches enable the study of neural correlates of subjective factors (such as a participant's beliefs about the reward to be received for performing some action) that the computational theories purport to quantify.     

Latent antigen vaccination in a model gammaherpesvirus infection

Vaccines that can reduce the load of latent gammaherpesvirus infections are eagerly sought. One attractive strategy is vaccination against latency-associated proteins, which may increase the efficiency with which T cells recognize and eliminate latently infected cells. However, due to the lack of tractable animal model systems, the effect of latent-antigen vaccination on gammaherpesvirus latency is not known. Here we use the murine gammaherpesvirus model to investigate the impact of vaccination with the latency-associated M2 antigen. As expected, vaccination had no effect on the acute lung infection. However, there was a significant reduction in the load of latently infected cells in the initial stages of the latent infection, when M2 is expressed. These data show for the first time that latent-antigen vaccination can reduce the level of latency in vivo and suggest that vaccination strategies involving other latent antigens may ultimately be successfully used to reduce the long-term latent infection.     

Haemodynamics and wall remodelling of a growing cerebral aneurysm: a computational model

We have developed a computational simulation model for investigating an often postulated hypothesis connected with aneurysm growth. This hypothesis involves a combination of two parallel and interconnected mechanisms: according to the first mechanism, an endothelium-originating and wall shear stress-driven apoptotic behavior of smooth muscle cells, leading to loss of vascular tone is believed to be important to the aneurysm behavior. Vascular tone refers to the degree of constriction experienced by a blood vessel relative to its maximally dilated state. All resistance and capacitance vessels under basal conditions exhibit some degree of smooth muscle contraction that determines the diameter, and hence tone, of the vessel. The second mechanism is connected to the arterial wall remodeling. Remodeling of the arterial wall under constant tension is a biomechanical process of rupture, degradation and reconstruction of the medial elastin and collagen fibers. In order to investigate these two mechanisms within a computationally tractable framework, we devise mechanical analogues that involve three-dimensional haemodynamics, yielding estimates of the wall shear stress and pressure fields and a quasi-steady approach for the apoptosis and remodeling of the wall. These analogues are guided by experimental information for the connection of stimuli to responses at a cellular level, properly averaged over volumes or surfaces. The model predicts aneurysm growth and can attribute specific roles to the two mechanisms involved: the smooth muscle cell-related loss of tone is important to the initiation of aneurysm growth, but cannot account alone for the formation of fully grown sacks; the fiber-related remodeling is pivotal for the latter.     

Hydrolysis of nerve agents by model nucleophiles: a computational study

Density functional theory calculations were employed to study the reaction of five nerve agents with model nucleophiles, including EtX(-) and EtXH (X=O, S, Se) for serine, cysteine and selenocysteine, respectively. Calculations at the B3LYP/6-311++G(2d,p) level of theory predict an exothermic reaction between ethoxide and all of the nerve agents studied. As compared to EtO(-) as a nucleophile, these reactions become approximately 30kcal/mol more endothermic for EtS(-), and by approximately 40kcal/mol for EtSe(-). The equivalent reactions with the neutral nucleophiles (EtXH) were more endothermic. The effect of solvation on the reaction thermochemistry was determined using a polarizable continuum model simulating the dielectric constant of chloroform. While there was a large exothermic shift for reactions involving charged nucleophiles with solvation modeling, the corresponding shift was minimal for the reaction with neutral nucleophiles.     

Venular endothelium-derived NO can affect paired arteriole: a computational model

Venular endothelial cells can release nitric oxide (NO) in response to intraluminal flow both in isolated venules and in vivo. Experimental studies suggest that venular endothelium-released NO causes dilation of the adjacent paired arteriole. In the vascular wall, NO stimulates its target hemoprotein, soluble guanylate cyclase (sGC), which relaxes smooth muscle cells. In this study, a computational model of NO transport for an arteriole and venule pair was developed to determine the importance of the venular endothelium-released NO and its transport to the adjacent arteriole in the tissue. The model predicts that the tissue NO levels are affected within a wide range of parameters, including NO-red blood cell reaction rate and NO production rate in the arteriole and venule. The results predict that changes in the venular NO production affected not only venular endothelial and smooth muscle NO concentration but also endothelial and smooth muscle NO concentration in the adjacent arteriole. This suggests that the anatomy of microvascular tissue can permit the transport of NO from arteriolar to venular side, and vice versa, and may provide a mechanism for dilation of proximal arterioles by venules. These results will have significant implications for our understanding of tissue NO levels in both physiological and pathophysiological conditions.     

Reactivation of model cholinesterases by oximes and intermediate phosphyloximes: a computational study

Phosphyloximes (POX) are generated upon the reactivation of organophosphorus (OP)-inhibited cholinesterases (ChEs) by pyridinium oximes. These POXs are known to be potent inhibitors of the ChEs following reactivation. However, they can also decompose to give an OP derivative and a cyano derivative of the oxime when a base abstracts the benzylic proton. Using density functional theory, thermodynamic properties were calculated for the reactivation and decomposition pathways of three different oximes (2-PAM, 3-PAM and 4-PAM) with six different OPs (cyclosarin, paraoxon, sarin, tabun, VR and VX). For reactivation purposes, 2-PAM is predicted to be more efficient than 3- and 4-PAM. Based on atomic charges and relative energies, 2-POXs were found to be more inclined towards the decomposition process.     

Development of a computational model for astronaut reorientation

The ability to model astronaut reorientations computationally provides a simple way to develop and study human motion control strategies. Since the cost of experimenting in microgravity is high, and underwater training can lead to motions inappropriate for microgravity, these techniques allow for motions to be developed and well-understood prior to any microgravity exposure. By including a model of the current space suit, we have the ability to study both intravehicular and extravehicular activities. We present several techniques for rotating about the axes of the body and show that motions performed by the legs create a greater net rotation than those performed by the arms. Adding a space suit to the motions was seen to increase the resistance torque and limit the available range of motion. While rotations about the body axes can be performed in the current space suit, the resulting motions generated a reduced rotation when compared to the unsuited configuration.