Modular inverse reinforcement learning for visuomotor behavior

In a large variety of situations one would like to have an expressive and accurate model of observed animal or human behavior. While general purpose mathematical models may capture successfully properties of observed behavior, it is desirable to root models in biological facts. Because of ample empirical evidence for reward-based learning in visuomotor tasks, we use a computational model based on the assumption that the observed agent is balancing the costs and benefits of its behavior to meet its goals. This leads to using the framework of reinforcement learning, which additionally provides well-established algorithms for learning of visuomotor task solutions. To quantify the agent’s goals as rewards implicit in the observed behavior, we propose to use inverse reinforcement learning, which quantifies the agent’s goals as rewards implicit in the observed behavior. Based on the assumption of a modular cognitive architecture, we introduce a modular inverse reinforcement learning algorithm that estimates the relative reward contributions of the component tasks in navigation, consisting of following a path while avoiding obstacles and approaching targets. It is shown how to recover the component reward weights for individual tasks and that variability in observed trajectories can be explained succinctly through behavioral goals. It is demonstrated through simulations that good estimates can be obtained already with modest amounts of observation data, which in turn allows the prediction of behavior in novel configurations.     

Influenza vaccine and learning in C57BL mice with an acute experimental autoimmune encephalomyelitis

Literature data suggest possible link between influenza vaccination and development of autoimmune processes. Therefore, the aim of the study was to investigate the effect of influenza vaccination on spatial learning in mice with experimental autoimmune encephalomyelitis (EAE). EAE was induced in eight-week-old C57BL/6J female mice by subcutaneous immunization (MOG_35–55 in complete Freund’s adjuvant) and Pertussis vaccine injected intraperitoneally. Mice were vaccinated with influenza vaccine three days before MOG immunization. The hippocampal-dependent spatial learning test, Morris Water Maze test (MWM), was performed before and after EAE induction. Significant difference (P < 0.05) in the time for completing the Morris Water Maze task was found between mice with mild clinical signs of EAE when compared to other mice. However no significant difference was observed between mice with EAE and mice with EAE that were vaccinated with influenza vaccine. Hippocampal tissue lesions in EAE mice are in correlation with memory impairment. Study shows no influence of influenza vaccine on progression of clinical signs of EAE, spatial learning and memory impairment.     

Perinatal BPA exposure demasculinizes males in measures of affect but has no effect on water maze learning in adulthood

Bisphenol A (BPA) is an endocrine disrupting agent that can alter the normal gonadal steroid-sensitive sexual differentiation of the brain and behavior. While reproductive behavior and physiology are known to be altered by perinatal exposure to this compound, less is known about BPA's effects on sex differences in learning and measures of affect. In order to evaluate the effects of perinatal BPA treatment on learning and affect in adulthood, we exposed rats to one of five doses of BPA through gestation and lactation then examined adult behavior in the Morris Water Maze (MWM), the Elevated Plus Maze (EPM) and the Forced Swim Test (FST). No effect of BPA was observed in the MWM, but on both the EPM and FST, low doses (5 μg/kg) of BPA eliminated sex differences found between controls; furthermore, a non-monotonic dose-response observed in previous studies was confirmed for these tasks. Overall, our study adds to the growing data suggesting that BPA interferes with the normal development of affective behaviors in a non-linear, dose-dependent manner.     

Forebrain-Specific Loss of BMPRII in Mice Reduces Anxiety and Increases Object Exploration

To investigate the role of Bone Morphogenic Protein Receptor Type II (BMPRII) in learning, memory, and exploratory behavior in mice, a tissue-specific knockout of BMPRII in the post-natal hippocampus and forebrain was generated. We found that BMPRII mutant mice had normal spatial learning and memory in the Morris water maze, but showed significantly reduced swimming speeds with increased floating behavior. Further analysis using the Porsolt Swim Test to investigate behavioral despair did not reveal any differences in immobility between mutants and controls. In the Elevated Plus Maze, BMPRII mutants and Smad4 mutants showed reduced anxiety, while in exploratory tests, BMPRII mutants showed more interest in object exploration. These results suggest that loss of BMPRII in the mouse hippocampus and forebrain does not disrupt spatial learning and memory encoding, but instead impacts exploratory and anxiety-related behaviors.     

Postnatal dexamethasone and long term learning and memory functions in developing rats: Effect of postnatal age and gender

In this study, we investigated the effect of dexamethasone on the long-term learning and memory functions in developing rats. In Sprague-Dawley rat pups, we administered a daily dose of dexamethasone (0.5 mg/kg/day) for three consecutive days in three groups of animals: the "ultra-early" group received steroids on postnatal days (PND) 1-3; the "early" group received the drug on PNDs 8-10, and the "late" group received the drug on PNDs 28-30. The control group was not given any medication. All animals underwent structured CNS examinations beginning on PND 15, and continued through PND 20. The pups were tested for spatial learning and memory functions using the Morris Water Maze (MWM) on PNDs 31 through 35, 45 through 49, and 59 through 63. They were also tested for reward-based learning and memory functions using Radial Arm Maze (RAM) on PNDs 70 through 72. We analyzed the effect of dexamethasone, postnatal age, and sex on neurological milestones, and learning and memory functions. We found that neurological examination findings were similar in all groups, as were the results of the reward-based learning using RAM. However, in the MWM, the total distance of swimming and the total time to find the hidden platform showed considerable difference among the groups. Although these functions improved with postnatal age, the female pups in all three steroid groups, and the male pups in the late-steroid group lagged significantly in learning and memory functions compared to the controls, and such lags were transient. However, the interaction terms between dexamethasone, age, and sex were also significant in MWM test results. Steroids administered postnatally may have transient, retarding effect on learning and memory functions, and that animal age and sex may modify such effects. Such lags are not global, but specific to the types of memory tests used, implicating different neural circuitries in the pathogenesis of such abnormalities. Although transient, if such adverse effects occur at critical phases during brain maturation, the implications for poor, long-term outcomes may be more significant. The mechanisms underlying such changes need to be explored.     

Dahl (S × R) Rat Congenic Strain Analysis Confirms and Defines a Chromosome 17 Spatial Navigation Quantitative Trait Locus to <10 Mbp

A quantitative trait locus (QTL) linked with ability to find a platform in the Morris Water Maze (MWM) was located on chromosome 17 (Nav-5 QTL) using intercross between Dahl S and Dahl R rats. We developed two congenic strains, S.R17A and S.R17B introgressing Dahl R-chromosome 17 segments into Dahl S chromosome 17 region spanning putative Nav-5 QTL. Performance analysis of S.R17A, S.R17B and Dahl S rats in the Morris water maze (MWM) task showed a significantly decreased spatial navigation performance in S.R17B congenic rats when compared with Dahl S controls (P = 0.02). The S.R17A congenic segment did not affect MWM performance delimiting Nav-5 to the chromosome 17 65.02–74.66 Mbp region. Additional fine mapping is necessary to identify the specific gene variant accounting for Nav-5 effect on spatial learning and memory in Dahl rats.     

Locating and navigation mechanism based on place-cell and grid-cell models

Extensive experiments on rats have shown that environmental cues play an important role in goal locating and navigation. Major studies about locating and navigation are carried out based only on place cells. Nevertheless, it is known that navigation may also rely on grid cells. Therefore, we model locating and navigation based on both, thus developing a novel grid-cell model, from which firing fields of grid cells can be obtained. We found a continuous-time dynamic system to describe learning and direction selection. In our simulation experiment, according to the results from physiology experiments, we successfully rebuild place fields of place cells and firing fields of grid cells. We analyzed the factors affecting the locating accuracy. Results show that the learning rate, firing threshold and cell number can influence the outcomes from various tasks. We used our system model to perform a goal navigation task and showed that paths that are changed for every run in one experiment converged to a stable one after several runs.     

Gender dimorphism in aspartame-induced impairment of spatial cognition and insulin sensitivity

Previous studies have linked aspartame consumption to impaired retention of learned behavior in rodents. Prenatal exposure to aspartame has also been shown to impair odor-associative learning in guinea pigs; and recently, aspartame-fed hyperlipidemic zebrafish exhibited weight gain, hyperglycemia and acute swimming defects. We therefore investigated the effects of chronic lifetime exposure to aspartame, commencing in utero, on changes in blood glucose parameters, spatial learning and memory in C57BL/6J mice. Morris Water Maze (MWM) testing was used to assess learning and memory, and a random-fed insulin tolerance test was performed to assess glucose homeostasis. Pearson correlation analysis was used to investigate the associations between body characteristics and MWM performance outcome variables. At 17 weeks of age, male aspartame-fed mice exhibited weight gain, elevated fasting glucose levels and decreased insulin sensitivity compared to controls (P<0.05). Females were less affected, but had significantly raised fasting glucose levels. During spatial learning trials in the MWM (acquisition training), the escape latencies of male aspartame-fed mice were consistently higher than controls, indicative of learning impairment. Thigmotactic behavior and time spent floating directionless was increased in aspartame mice, who also spent less time searching in the target quadrant of the maze (P<0.05). Spatial learning of female aspartame-fed mice was not significantly different from controls. Reference memory during a probe test was affected in both genders, with the aspartame-fed mice spending significantly less time searching for the former location of the platform. Interestingly, the extent of visceral fat deposition correlated positively with non-spatial search strategies such as floating and thigmotaxis, and negatively with time spent in the target quadrant and swimming across the location of the escape platform. These data suggest that lifetime exposure to aspartame, commencing in utero, may affect spatial cognition and glucose homeostasis in C57BL/6J mice, particularly in males.     

Mobile phone exposure and spatial memory

Radiofrequency (RF) emission during mobile phone use has been suggested to impair cognitive functions, that is, working memory. This study investigated the effects of a 2 1/2 h RF exposure (884 MHz) on spatial memory and learning, using a double-blind repeated measures design. The exposure was designed to mimic that experienced during a real-life mobile phone conversation. The design maximized the exposure to the left hemisphere. The average exposure was peak spatial specific absorption rate (psSAR10g) of 1.4 W/kg. The primary outcome measure was a "virtual" spatial navigation task modeled after the commonly used and validated Morris Water Maze. The distance traveled on each trial and the amount of improvement across trials (i.e., learning) were used as dependent variables. The participants were daily mobile phone users, with and without symptoms attributed to regular mobile phone use. Results revealed a main effect of RF exposure and a significant RF exposure by group effect on distance traveled during the trials. The symptomatic group improved their performance during RF exposure while there was no such effect in the non-symptomatic group. Until this new finding is further investigated, we can only speculate about the cause.     

Evidence for hippocampal role in place avoidance other than merely memory storage

Spatial navigation is used as a popular animal model of higher cognitive functions in people. The data suggest that the hippocampus is important for both storing spatial memories and for performing spatial computations necessary for navigation. Animals use multiple behavioral strategies to solve spatial tasks often using multiple memory systems. We investigated how inactivation of the rat hippocampus affects performance in a place avoidance task to determine if the role of the hippocampus in this task could be attributed to memory storage/retrieval or to the computations needed for navigation. Injecting tetrodotoxin (TTX) into both hippocampi impaired conditioned place avoidance, but after injecting only one hippocampus, the rats learned the place avoidance as well as without any injections. Retention of the place avoidance learned with one hippocampus was not impaired when the injection was switched to the hippocampus that had not been injected during learning. The result suggests that during learning, the hippocampus did not store the place avoidance memory.     

Morris Water Maze Test for Learning and Memory Deficits in Alzheimer's Disease Model Mice

The Morris Water Maze (MWM) was first established by neuroscientist Richard G. Morris in 1981 in order to test hippocampal-dependent learning, including acquisition of spatial memoryand long-term spatial memory ¹. The MWM is a relatively simple procedure typically consisting of six day trials, the main advantage being the differentiation between the spatial (hidden-platform) and non-spatial (visible platform) conditions ^2-4. In addition, the MWM testing environment reduces odor trail interference ⁵. This has led the task to be used extensively in the study of the neurobiology and neuropharmacology of spatial learning and memory. The MWM plays an important role in the validation of rodent models for neurocognitive disorders such as Alzheimer’s Disease ^{6, 7}. In this protocol we discussed the typical procedure of MWM for testing learning and memory and data analysis commonly used in Alzheimer’s disease transgenic model mice.     

Odor supported place cell model and goal navigation in rodents

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation.     

The dopamine circuit as a reward-taxis navigation system

Studying the brain circuits that control behavior is challenging, since in addition to their structural complexity there are continuous feedback interactions between actions and sensed inputs from the environment. It is therefore important to identify mathematical principles that can be used to develop testable hypotheses. In this study, we use ideas and concepts from systems biology to study the dopamine system, which controls learning, motivation, and movement. Using data from neuronal recordings in behavioral experiments, we developed a mathematical model for dopamine responses and the effect of dopamine on movement. We show that the dopamine system shares core functional analogies with bacterial chemotaxis. Just as chemotaxis robustly climbs chemical attractant gradients, the dopamine circuit performs ‘reward-taxis’ where the attractant is the expected value of reward. The reward-taxis mechanism provides a simple explanation for scale-invariant dopaminergic responses and for matching in free operant settings, and makes testable quantitative predictions. We propose that reward-taxis is a simple and robust navigation strategy that complements other, more goal-directed navigation mechanisms.     

Spatial memory tasks in rodents: what do they model?

The analysis of spatial learning and memory in rodents is commonly used to investigate the mechanisms underlying certain forms of human cognition and to model their dysfunction in neuropsychiatric and neurodegenerative diseases. Proper interpretation of rodent behavior in terms of spatial memory and as a model of human cognitive functions is only possible if various navigation strategies and factors controlling the performance of the animal in a spatial task are taken into consideration. The aim of this review is to describe the experimental approaches that are being used for the study of spatial memory in rats and mice and the way that they can be interpreted in terms of general memory functions. After an introduction to the classification of memory into various categories and respective underlying neuroanatomical substrates, I explain the concept of spatial memory and its measurement in rats and mice by analysis of their navigation strategies. Subsequently, I describe the most common paradigms for spatial memory assessment with specific focus on methodological issues relevant for the correct interpretation of the results in terms of cognitive function. Finally, I present recent advances in the use of spatial memory tasks to investigate episodic-like memory in mice.     

Socially biased learning among adult cottontop tamarins (Saguinus oedipus)

We presented adult cottontop tamarins (Saguinus oedipus) with a novel foraging task that had been used previously to examine socially biased learning of juvenile observers [Humle & Snowdon, Animal Behaviour 75:267–277, 2008]. The task could be solved in one of two ways, and thus allowed for an analysis of behavioral matching between an observer and a skilled demonstrator (trained to use one of the two methods exclusively). Because the demonstrator was an adult in both this study and the juvenile study, the influence of the observer's age could be isolated and examined, as well as the behavior of demonstrators toward observers of different ages. Our main goals were to (1) compare adults and juveniles acquiring the same task to identify how the age of the observer affects socially biased learning and (2) examine the relationship between socially biased learning and behavioral matching in adults. Although adults spent less time observing the trained demonstrators than did juveniles, the adults were more proficient at solving the task. Furthermore, even though observers did not overtly match the behavior of the demonstrator, observation remained an important factor in the success of these individuals. The findings suggested that adult observers could extract information needed to solve a novel foraging task without explicitly matching the behavior of the demonstrator. Adult observers begged much less than juveniles and demonstrators did not respond to begging from adult. Skill acquisition and the process of socially biased learning are, therefore, age‐dependent and are influenced by the behavioral interactions between observer and demonstrator. To what extent this holds true for other primates or animal species still needs to be more fully investigated and considered when designing experiments and interpreting results. Am. J. Primatol. 72:287–295, 2010. © 2009 Wiley‐Liss, Inc.     

Ca2+ spikes in the flagellum control chemotactic behavior of sperm

The events that occur during chemotaxis of sperm are only partly known. As an essential step toward determining the underlying mechanism, we have recorded Ca2+ dynamics in swimming sperm of marine invertebrates. Stimulation of the sea urchin Arbacia punctulata by the chemoattractant or by intracellular cGMP evokes Ca2+ spikes in the flagellum. A Ca2+ spike elicits a turn in the trajectory followed by a period of straight swimming ('turn-and-run'). The train of Ca2+ spikes gives rise to repetitive loop-like movements. When sperm swim in a concentration gradient of the attractant, the Ca2+ spikes and the stimulus function are synchronized, suggesting that precise timing of Ca2+ spikes controls navigation. We identified the peptide asterosap as a chemotactic factor of the starfish Asterias amurensis. The Ca2+ spikes and swimming behavior of sperm from starfish and sea urchin are similar, implying that the signaling pathway of chemotaxis has been conserved for almost 500 million years.     

The neuronal basis of behavior in Tritonia. III. Neuronal mechanism of a fixed action pattern

We have investigated the roles played by numerous identified brain cells in initiating and controlling the coordinated sequence of movements of an instinctive escape-swimming sequence in an intact animal preparation of the nudibranch mollusc Tritonia diomedia. Intracellular electrical activity in different neurons has been correlated with the various phases of the behavior. We recognized four major stages in the response: (1) reflex local withdrawal; (2) preparation for swimming; (3) swimming; and (4) termination. We have located and studied brain cells whose activity is associated with the following aspects of swimming: withdrawal; elongation; triggering behavior; dorsal flexion; ventral flexion; and neurons which excite both dorsal and ventral flexor neurons simulataneously. We find that specific neurons play clearly defined and invariant roles in control of escape-swimming and that the neuronal circuitry underlying the coordination of the sequence is the same in different individuals of the species. Details of the neuronal circuitry and a number of the general functional attributes of interacting cell groups have been determined directly or inferred from observations of cell to cell interactions. A preliminary model of the neuronal apparatus which controls this behavior is discussed. The principal findings are: (1) a discrete group of electrically coupled neurons determines, by its output, whether or not escapeswimming will be executed; (2) the neuronal elements responsible for execution of the swimming stages of the sequence are maintained in an excited state for the required period, in part by a regenerative feedback system; (3) alternating bursts of impulses in functional antagonists are co-ordinated in part by reciprocal inhibition between them; and (4) termination of the sequence occurs abruptly at a particular phase in the swimming cycle and appears to be an active neural process, rather than a simple running-down.     

色氨酸对应激小鼠行为学和脑内单胺类递质的影响

本文以小鼠为实验材料建模,依次进行了Morris水迷宫、跳台、强迫游泳实验。结果显示,应激可导致小鼠学习记忆能力下降,容易出现绝望情绪,下丘脑5-HT含量下降,NE含量上升,灌胃高低剂量色氨酸(100,200mg·kg-1 bw)均可显著提高小鼠学习记忆能力,缓和绝望情绪。低剂量色氨酸在增加下丘脑5-羟色胺(5-HT)含量和降低去甲肾上腺素(NE)含量方面效果更显著。结果表明,适量色氨酸可以有效改善应激小鼠相关行为学评分,其机制可能与下丘脑5-HT和NE的含量有关。     

Editing of the serotonin 2C receptor pre-mRNA: Effects of the Morris Water Maze

The pre-mRNA encoding the serotonin 2C receptor, HTR2C (official mouse gene symbol, Htr2c), is subject to adenosine deamination that produces inosine at five sites within the coding region. Combinations of this site-specific A-to-I editing can produce 32 different mRNA sequences encoding 24 different protein isoforms with differing biochemical and pharmacological properties. Studies in humans have reported abnormalities in patterns of HTR2C editing in psychiatric disorders, and studies in rodents show altered patterns of editing in response to drug treatments and stressful situations. To further explore the biological significance of editing of the Htr2c mRNA and its regulation, we have examined patterns of Htr2c editing in C57BL/6J mice after exposure to the hidden platform version of the Morris Water Maze, a test of spatial learning that, in mice, is also associated with stress. In brains of both swimming controls and mice trained to find the platform, subtle time dependent changes in editing patterns are seen as soon as 1 h after a probe trial and typically last less than 24 h. Changes in whole brain with cerebellum removed differ from those seen in isolated hippocampus and cortex. Unexpectedly, in hippocampi from subsets of mice, abnormally low levels of editing were seen that were not correlated with behavior or with editing levels in cortex. These data implicate responses to spatial learning and stress, in addition to stochastic processes, in the generation of subtle changes in editing patterns of Htr2c.     

Motor learning with unstable neural representations

It is often assumed that learning takes place by changing an otherwise stable neural representation. To test this assumption, we studied changes in the directional tuning of primate motor cortical neurons during reaching movements performed in familiar and novel environments. During the familiar task, tuning curves exhibited slow random drift. During learning of the novel task, random drift was accompanied by systematic shifts of tuning curves. Our analysis suggests that motor learning is based on a surprisingly unstable neural representation. To explain these results, we propose that motor cortex is a redundant neural network, i.e., any single behavior can be realized by multiple configurations of synaptic strengths. We further hypothesize that synaptic modifications underlying learning contain a random component, which causes wandering among synaptic configurations with equivalent behaviors but different neural representations. We use a simple model to explore the implications of these assumptions.