Embodied semantics for actions: findings from functional brain imaging.

The theory of embodied semantics for actions specifies that the sensory-motor areas used for producing an action are also used for the conceptual representation of the same action. Here we review the functional imaging literature that has explored this theory and consider both supporting as well as challenging fMRI findings. In particular we address the representation of actions and concepts as well as literal and metaphorical phrases in the premotor cortex.     

Differences in rat dorsal striatal NMDA and AMPA receptors following acute and repeated cocaine-induced locomotor activation

Sprague-Dawley rats can be classified as low or high cocaine responders (LCRs or HCRs, respectively) based on their locomotor activity induced by an acute low dose of cocaine. Upon repeated cocaine exposure, LCRs display greater locomotor sensitization, reward, and reinforcement than HCRs. Altered glutamate receptor expression in the brain reward pathway has been linked to locomotor sensitization and addiction. To determine if such changes contribute to the differential development of locomotor sensitization, we examined protein levels of total, phosphorylated, and cell surface glutamate N-methyl D-aspartate (NMDA) and α-amino-3-hydroxy-5-methylisoxazole-4-propionate (AMPA) receptors (Rs) following acute or repeated cocaine (10 mg/kg, i.p.) in LCRs, HCRs and saline controls. Three areas involved in the development and expression of locomotor sensitization were investigated: the ventral tegmental area (VTA), nucleus accumbens (NAc) and dorsal striatum (dSTR). Our results revealed differences only in the dSTR, where we found that after acute cocaine, GluN2B(Tyr-1472) phosphorylation was significantly greater in LCRs, compared to HCRs and controls. Additionally in dSTR, after repeated cocaine, we observed significant increases in total GluA1, phosphorylated GluA1(Ser-845), and cell surface GluA1 in all cocaine-treated animals vs. controls. The acute cocaine-induced increases in NMDARs in dSTR of LCRs may help to explain the more ready development of locomotor sensitization and susceptibility to addiction-like behaviors in rats that initially exhibit little or no cocaine-induced activation, whereas the AMPAR increases after repeated cocaine may relate to recruitment of more dorsal striatal circuits and maintenance of the marked cocaine-induced locomotor activation observed in all of the rats.     

A theory of cerebral learning regulated by the reward system

Hypothetical mechanisms of the neocorticohippocampal system are presented. Neurophysiological and neuroanatomical findings concerning the system are integrated to demonstrate how animals associate sensory stimuli with rewarding actions: (1) cortical plasticity regulated by cholinergic/noradrenergic inputs from the hypothalamic reward system reinforces association connections between the most activated columns in the cortex; (2) the repetitive reinforcement forms association pathways connecting sensory cortical columns activated by the stimuli with motor cortical columns producing the rewarding actions; (3) after the pathways are formed, the cortex is capable of temporarily memorizing the stimuli by producing long-term potentiation through the cortico-hippocampal circuits; and (4) the memory allows the cortex to extend correct association pathways even in an environment where sensory stimuli rapidly change. A mathematical model of parts of the nervous system is presented to quantitatively examine the mechanisms. Membrane characteristics of single neurons are given by the Hodgkin-Huxley electric circuit. According to anatomical data, neural circuits of the neocortico-hippocampal system are composed by connecting populations of the model neurons. Computer simulation using physiological data concerning ion channels demonstrates how the mechanisms work and how to test the hypotheses presented.A preliminary version of this paper was presented at the Fifteenth Annual Meeting of the Japan Neuroscience Society, Tokyo, December 1991     

The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning

While there is no doubt that social signals affect human reinforcement learning, there is still no consensus about how this process is computationally implemented. To address this issue, we compared three psychologically plausible hypotheses about the algorithmic implementation of imitation in reinforcement learning. The first hypothesis, decision biasing (DB), postulates that imitation consists in transiently biasing the learner’s action selection without affecting their value function. According to the second hypothesis, model-based imitation (MB), the learner infers the demonstrator’s value function through inverse reinforcement learning and uses it to bias action selection. Finally, according to the third hypothesis, value shaping (VS), the demonstrator’s actions directly affect the learner’s value function. We tested these three hypotheses in 2 experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task. We show through model comparison and model simulation that VS provides the best explanation of learner’s behavior. Results replicated in a third independent experiment featuring a larger cohort and a different design (N = 302). In our experiments, we also manipulated the quality of the demonstrators’ choices and found that learners were able to adapt their imitation rate, so that only skilled demonstrators were imitated. We proposed and tested an efficient meta-learning process to account for this effect, where imitation is regulated by the agreement between the learner and the demonstrator. In sum, our findings provide new insights and perspectives on the computational mechanisms underlying adaptive imitation in human reinforcement learning.

This study investigates imitation from a computational perspective; three experiments show that, in the context of reinforcement learning, imitation operates via a durable modification of the learner''s values, shedding new light on how imitation is computationally implemented and shapes learning and decision-making.     

Anterior prefrontal cortex inhibition impairs control over social emotional actions

When dealing with emotional situations, we often need to rapidly override automatic stimulus-response mappings and select an alternative course of action [1], for instance, when trying to manage, rather than avoid, another's aggressive behavior. The anterior prefrontal cortex (aPFC) has been linked to the control of these social emotional behaviors [2, 3]. We studied how this control is implemented by inhibiting the left aPFC with continuous theta burst stimulation (cTBS; [4]). The behavioral and cerebral consequences of this intervention were assessed with a task quantifying the control of social emotional actions and with concurrent measurements of brain perfusion. Inhibition of the aPFC led participants to commit more errors when they needed to select rule-driven responses overriding automatic action tendencies evoked by emotional faces. Concurrently, task-related perfusion decreased in bilateral aPFC and posterior parietal cortex and increased in amygdala and left fusiform face area. We infer that the aPFC controls social emotional behavior by upregulating regions involved in rule selection [5] and downregulating regions supporting the automatic evaluation of emotions [6]. These findings illustrate how exerting emotional control during social interactions requires the aPFC to coordinate rapid action selection processes, the detection of emotional conflicts, and the inhibition of emotionally-driven responses.     

Viewing lip forms: cortical dynamics

Viewing other persons' actions automatically activates brain areas belonging to the mirror-neuron system (MNS) assumed to link action execution and observation. We followed, by magnetoencephalographic cortical dynamics, subjects who observed still pictures of lip forms, on-line imitated them, or made similar forms in a self-paced manner. In all conditions and in both hemispheres, cortical activation progressed in 20-70 ms steps from the occipital cortex to the superior temporal region (where the strongest activation took place), the inferior parietal lobule, and the inferior frontal lobe (Broca's area), and finally, 50-140 ms later, to the primary motor cortex. The signals of Broca's area and motor cortex were significantly stronger during imitation than other conditions. These results demonstrate that still pictures, only implying motion, activate the human MNS in a well-defined temporal order.     

CAN GENE FLOW PREVENT REINFORCEMENT?

A model of reinforcement in a hybrid zone is developed in which an allele causing reinforcement may only be favored in the center of the hybrid zone and is selected against elsewhere. When reinforcement is favored only in the hybrid zone, the swamping effect of gene flow severely impedes the evolution of reinforcement: reinforcement can only occur when β < sγ²/4 (s is selection against hybrids, γ is the strength of reinforcement, and β is the strength of selection against reinforcement). Although the region in which reinforcement can occur is narrower when selection against hybrids is stronger, strong selection is still more likely to be reinforced despite the effects of gene flow. Another modifier is considered which reduces postmating isolation. The conditions for increase in frequency of this modifier reduce to the same conditions as those for the reinforcing allele. Both kinds of modification reduce the strength of selection and, therefore, cause the cline to widen. Reinforcement causes an increase in premating isolation between races, while modification of selection reduces postmating isolation.     

A memory system in the monkey

A neural model is presented, based largely on evidence from studies in monkeys, postulating that coded representation of stimuli are stored in the higher-order sensory (i.e. association) areas of the cortex whenever stimulus activation of these areas also triggers a cortico-limbo-thalamo-cortical circuit. This circuit, which could act as either an imprinting or rehearsal mechanism, may actually consist of two parallel circuits, one involving the amygdala and the dorsomedial nucleus of the thalamus, and the other the hippocampus and the anterior nuclei. The stimulus representation stored in cortex by action of these circuits is seen as mediating three different memory processes: recognition, which occurs when the stored representation is reactivated via the original sensory pathway; recall, when it is reactivated via any other pathway; and association, when it activates other stored representations (sensory, affective, spatial, motor) via the outputs of the higher-order sensory areas to the relevant structures.     

Nicotinic modulation of synaptic transmission and plasticity in cortico-limbic circuits

Nicotine is the principle addictive agent delivered via cigarette smoking. The addictive activity of nicotine is due to potent interactions with nicotinic acetylcholine receptors (nAChRs) on neurons in the reinforcement and reward circuits of the brain. Beyond its addictive actions, nicotine is thought to have positive effects on performance in working memory and short-term attention-related tasks. The brain areas involved in such behaviors are part of an extensive cortico-limbic network that includes relays between prefrontal cortex (PFC) and cingulate cortex (CC), hippocampus, amygdala, ventral tegmental area (VTA) and the nucleus accumbens (nAcc). Nicotine activates a broad array of nAChRs subtypes that can be targeted to pre- as well as peri- and post-synaptic locations in these areas. Thereby, nicotine not only excites different types of neurons, but it also perturbs baseline neuronal communication, alters synaptic properties and modulates synaptic plasticity.In this review we focus on recent findings on nicotinic modulation of cortical circuits and their targets fields, which show that acute and transient activation of nicotinic receptors in cortico-limbic circuits triggers a series of events that affects cognitive performance in a long lasting manner. Understanding how nicotine induces long-term changes in synapses and alters plasticity in the cortico-limbic circuits is essential to determining how these areas interact in decoding fundamental aspects of cognition and reward.     

Two-phase model of the basal ganglia: implications for discontinuous control of the motor system

In this article, I point out that simple one-phase models of the role of the basal ganglia in action selection have a problem. Furthermore, I suggest a solution with major implications for the organization of the action-selection and motor systems. In current models, the striatum evaluates multiple potential actions by adding biases based on previous conditioning. These biases may arise in both the direct (bias for) and indirect (bias against) pathways. Together, these biases influence which action is ultimately chosen. For efficient conditioning to occur, a positive outcome must selectively strengthen the striatal bias for the chosen action (via a dopaminergic mechanism). This is problematic, however, because all potential action choices have influenced firing patterns in striatal cells during the selection process; it is therefore unclear how the synapses that represent the chosen plan could be selectively strengthened. I suggest a simple solution in which the striatum has two functional phases. In the first phase, the basal ganglia provide biases for multiple potential actions (using both the direct and indirect pathways), leading to the choice of a single action in the cortex. In the second phase, an efference copy of the chosen action is sent to the striatum, where it contributes to the establishment of the eligibility trace for that action. This trace, when acted on by subsequent dopaminergic reinforcement, leads to specific strengthening of the bias only for the chosen action. Consistent with this model, recordings show post-choice imposition onto the striatum of signals corresponding to the chosen action. The existence of dual phases of basal ganglia function implies that decisions about action choice are sent to the motor system in a discontinuous manner. This would not be problematic if the motor system also operated discontinuously. I will review evidence suggesting that this is the case, notably that action is organized by approximately 10 Hz oscillations.     

Plasticity in the Rat Prefrontal Cortex: Linking Gene Expression and an Operant Learning with a Computational Theory

The plasticity in the medial Prefrontal Cortex (mPFC) of rodents or lateral prefrontal cortex in non human primates (lPFC), plays a key role neural circuits involved in learning and memory. Several genes, like brain-derived neurotrophic factor (BDNF), cAMP response element binding (CREB), Synapsin I, Calcium/calmodulin-dependent protein kinase II (CamKII), activity-regulated cytoskeleton-associated protein (Arc), c-jun and c-fos have been related to plasticity processes. We analysed differential expression of related plasticity genes and immediate early genes in the mPFC of rats during learning an operant conditioning task. Incompletely and completely trained animals were studied because of the distinct events predicted by our computational model at different learning stages. During learning an operant conditioning task, we measured changes in the mRNA levels by Real-Time RT-PCR during learning; expression of these markers associated to plasticity was incremented while learning and such increments began to decline when the task was learned. The plasticity changes in the lPFC during learning predicted by the model matched up with those of the representative gene BDNF. Herein, we showed for the first time that plasticity in the mPFC in rats during learning of an operant conditioning is higher while learning than when the task is learned, using an integrative approach of a computational model and gene expression.     

Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system

The prefrontal cortex (PFC) has long been thought to serve as an 'executive' that controls the selection of actions and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified often amounting to a homunculus. This paper reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a 'Go' versus 'NoGo' modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.     

Cortical mechanisms of action selection: the affordance competition hypothesis

At every moment, the natural world presents animals with two fundamental pragmatic problems: selection between actions that are currently possible and specification of the parameters or metrics of those actions. It is commonly suggested that the brain addresses these by first constructing representations of the world on which to build knowledge and make a decision, and then by computing and executing an action plan. However, neurophysiological data argue against this serial viewpoint. In contrast, it is proposed here that the brain processes sensory information to specify, in parallel, several potential actions that are currently available. These potential actions compete against each other for further processing, while information is collected to bias this competition until a single response is selected. The hypothesis suggests that the dorsal visual system specifies actions which compete against each other within the fronto-parietal cortex, while a variety of biasing influences are provided by prefrontal regions and the basal ganglia. A computational model is described, which illustrates how this competition may take place in the cerebral cortex. Simulations of the model capture qualitative features of neurophysiological data and reproduce various behavioural phenomena.     

Motor-sensory recalibration leads to an illusory reversal of action and sensation

To judge causality, organisms must determine the temporal order of their actions and sensations. However, this judgment may be confounded by changing delays in sensory pathways, suggesting the need for dynamic temporal recalibration. To test for such a mechanism, we artificially injected a fixed delay between participants' actions (keypresses) and subsequent sensations (flashes). After participants adapted to this delay, flashes at unexpectedly short delays after the keypress were often perceived as occurring before the keypress, demonstrating a recalibration of motor-sensory temporal order judgments. When participants experienced illusory reversals, fMRI BOLD signals increased in anterior cingulate cortex/medial frontal cortex (ACC/MFC), a brain region previously implicated in conflict monitoring. This illusion-specific activation suggests that the brain maintains not only a recalibrated representation of timing, but also a less-plastic representation against which to compare it.     

Mirroring and the development of action understanding

The discovery of mirror neurons in the monkey motor cortex has inspired wide-ranging hypotheses about the potential relationship between action control and social cognition. In this paper, we consider the hypothesis that this relationship supports the early development of a critical aspect of social understanding, the ability to analyse others’ actions in terms of goals. Recent investigations of infant action understanding have revealed rich connections between motor development and the analysis of goals in others’ actions. In particular, infants’ own goal-directed actions influence their analysis of others’ goals. This evidence indicates that the cognitive systems that drive infants’ own actions contribute to their analysis of goals in others’ actions. These effects occur at a relatively abstract level of analysis both in terms of the structure infants perceive in others’ actions and relevant structure in infants’ own actions. Although the neural bases of these effects in infants are not yet well understood, current evidence indicates that connections between action production and action perception in infancy involve the interrelated neural systems at work in generating planned, intelligent action.     

Expectations and outcomes: decision-making in the primate brain

Success in a constantly changing environment requires that decision-making strategies be updated as reward contingencies change. How this is accomplished by the nervous system has, until recently, remained a profound mystery. New studies coupling economic theory with neurophysiological techniques have revealed the explicit representation of behavioral value. Specifically, when fluid reinforcement is paired with visually-guided eye movements, neurons in parietal cortex, prefrontal cortex, the basal ganglia, and superior colliculus—all nodes in a network linking visual stimulation with the generation of oculomotor behavior—encode the expected value of targets lying within their response fields. Other brain areas have been implicated in the processing of reward-related information in the abstract: midbrain dopaminergic neurons, for instance, signal an error in reward prediction. Still other brain areas link information about reward to the selection and performance of specific actions in order for behavior to adapt to changing environmental exigencies. Neurons in posterior cingulate cortex have been shown to carry signals related to both reward outcomes and oculomotor behavior, suggesting that they participate in updating estimates of orienting value.     

Properties of Neurons in External Globus Pallidus Can Support Optimal Action Selection

The external globus pallidus (GPe) is a key nucleus within basal ganglia circuits that are thought to be involved in action selection. A class of computational models assumes that, during action selection, the basal ganglia compute for all actions available in a given context the probabilities that they should be selected. These models suggest that a network of GPe and subthalamic nucleus (STN) neurons computes the normalization term in Bayes’ equation. In order to perform such computation, the GPe needs to send feedback to the STN equal to a particular function of the activity of STN neurons. However, the complex form of this function makes it unlikely that individual GPe neurons, or even a single GPe cell type, could compute it. Here, we demonstrate how this function could be computed within a network containing two types of GABAergic GPe projection neuron, so-called ‘prototypic’ and ‘arkypallidal’ neurons, that have different response properties in vivo and distinct connections. We compare our model predictions with the experimentally-reported connectivity and input-output functions (f-I curves) of the two populations of GPe neurons. We show that, together, these dichotomous cell types fulfil the requirements necessary to compute the function needed for optimal action selection. We conclude that, by virtue of their distinct response properties and connectivities, a network of arkypallidal and prototypic GPe neurons comprises a neural substrate capable of supporting the computation of the posterior probabilities of actions.     

Influence of budget and reinforcement location on risk-sensitive preference

Little is known about the effect that procedural variables have on risk-sensitive preference. This study assessed the effect of procedural variables on pigeons' choice between a fixed and variable amount of reinforcement (amount risk) and, in a separate condition, between a fixed and variable delay until reinforcement (delay risk). Experiment 1 investigated the impact of water reinforcement and risk dimension when pigeons were in a restrictive budget, where access to water was less than that necessary to maintain current body weight, and a condition where the pigeons had ample access to water. Pigeons exhibited a greater tendency to prefer the variable alternative for delay risk than for amount risk in both restrictive and ample budgets. Varying water budget had no effect on risk preference. Experiment 2 investigated the influence of water reinforcer location while in a restrictive budget, in which reinforcers were delivered to a single location, two distinct locations, or a randomly selected location. With amount risk, pigeons were risk averse when reinforcers were delivered in separate or random locations and were indifferent to risk when delivered to a single location. With delay risk, pigeons were generally risk prone with no effect from reinforcement location. The finding that pigeons were risk averse when reinforcers were delivered to separate locations and were indifferent to risk when delivered to a single location offers a methodological explanation to the inconsistent findings in the literature with amount risk.     

Frontal cortex and reward-guided learning and decision-making

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.