Macaque Monkeys Can Learn Token Values from Human Models through Vicarious Reward

Monkeys can learn the symbolic meaning of tokens, and exchange them to get a reward. Monkeys can also learn the symbolic value of a token by observing conspecifics but it is not clear if they can learn passively by observing other actors, e.g., humans. To answer this question, we tested two monkeys in a token exchange paradigm in three experiments. Monkeys learned token values through observation of human models exchanging them. We used, after a phase of object familiarization, different sets of tokens. One token of each set was rewarded with a bit of apple. Other tokens had zero value (neutral tokens). Each token was presented only in one set. During the observation phase, monkeys watched the human model exchange tokens and watched them consume rewards (vicarious rewards). In the test phase, the monkeys were asked to exchange one of the tokens for food reward. Sets of three tokens were used in the first experiment and sets of two tokens were used in the second and third experiments. The valuable token was presented with different probabilities in the observation phase during the first and second experiments in which the monkeys exchanged the valuable token more frequently than any of the neutral tokens. The third experiments examined the effect of unequal probabilities. Our results support the view that monkeys can learn from non-conspecific actors through vicarious reward, even a symbolic task like the token-exchange task.     

Perceptual classification in a rapidly changing environment

Humans and monkeys can learn to classify perceptual information in a statistically optimal fashion if the functional groupings remain stable over many hundreds of trials, but little is known about categorization when the environment changes rapidly. Here, we used a combination of computational modeling and functional neuroimaging to understand how humans classify visual stimuli drawn from categories whose mean and variance jumped unpredictably. Models based on optimal learning (Bayesian model) and a cognitive strategy (working memory model) both explained unique variance in choice, reaction time, and brain activity. However, the working memory model was the best predictor of performance in volatile environments, whereas statistically optimal performance emerged in periods of relative stability. Bayesian and working memory models predicted decision-related activity in distinct regions of the prefrontal cortex and midbrain. These findings suggest that perceptual category judgments, like value-guided choices, may be guided by multiple controllers.     

Prefrontal cortex activity related to abstract response strategies

Many monkeys adopt abstract response strategies as they learn to map visual symbols to responses by trial and error. According to the repeat-stay strategy, if a symbol repeats from a previous, successful trial, the monkeys should stay with their most recent response choice. According to the change-shift strategy, if the symbol changes, the monkeys should shift to a different choice. We recorded the activity of prefrontal cortex neurons while monkeys chose responses according to these two strategies. Many neurons had activity selective for the strategy used. In a subsequent block of trials, the monkeys learned fixed stimulus-response mappings with the same stimuli. Some neurons had activity selective for choosing responses based on fixed mappings, others for choosing based on abstract strategies. These findings indicate that the prefrontal cortex contributes to the implementation of the abstract response strategies that monkeys use during trial-and-error learning.     

Specificity of sensorimotor learning and the neural code: Neuronal representations in the primary motor cortex

Human studies show that the learning of a new sensorimotor mapping that requires adaptation to directional errors is local and generalizes poorly to untrained directions. We trained monkeys to learn new visuomotor rotations for only one target in space and recorded neuronal activity in the primary motor cortex before, during and after learning. Similar to humans, the monkeys showed poor transfer of learning to other directions, as observed by behavioral aftereffects for untrained directions. To test for internal representations underlying these changes, we compared two features of neuronal activity before and after learning: changes in firing rates and changes in information content. Specific elevations of firing rate were only observed in a subpopulation of cells in the motor cortex with directional properties corresponding to the locally learned rotation; namely cells only showed plasticity if their preferred direction was near the training one. We applied measures from information theory to probe for learning-related changes in the neuronal code. Single cells conveyed more information about the direction of movement and this specific improvement in encoding was correlated with an increase in the slope of the neurons' tuning curve. Further, the improved information after learning enabled a more accurate reconstruction of movement direction from neuronal populations. Our findings suggest a neural mechanism for the confined generalization of a newly acquired internal model by showing a tight relationship between the locality of learning and the properties of neurons. They also provide direct evidence for improvement in the neural code as a result of learning.     

Vorbildwahl in der Gesangsentwicklung beim Japanischen Mövchen (Loncbura striata var. domestica,Estrildidae)

Birds which must learn their species-specific song need a means for choosing the appropriate song model. As individual Bengalese Finch ♂♂ have distinctive songs and song elements it is possible to determine a juvenile's choice of song models from within a restricted population. The results suggest that after an early period of learning from several models, juveniles preferentially copy the song of their father.     

From Birdsong to Human Speech Recognition: Bayesian Inference on a Hierarchy of Nonlinear Dynamical Systems

Our knowledge about the computational mechanisms underlying human learning and recognition of sound sequences, especially speech, is still very limited. One difficulty in deciphering the exact means by which humans recognize speech is that there are scarce experimental findings at a neuronal, microscopic level. Here, we show that our neuronal-computational understanding of speech learning and recognition may be vastly improved by looking at an animal model, i.e., the songbird, which faces the same challenge as humans: to learn and decode complex auditory input, in an online fashion. Motivated by striking similarities between the human and songbird neural recognition systems at the macroscopic level, we assumed that the human brain uses the same computational principles at a microscopic level and translated a birdsong model into a novel human sound learning and recognition model with an emphasis on speech. We show that the resulting Bayesian model with a hierarchy of nonlinear dynamical systems can learn speech samples such as words rapidly and recognize them robustly, even in adverse conditions. In addition, we show that recognition can be performed even when words are spoken by different speakers and with different accents—an everyday situation in which current state-of-the-art speech recognition models often fail. The model can also be used to qualitatively explain behavioral data on human speech learning and derive predictions for future experiments.     

适用于鼠学习记忆行为训练的三个新模型

目的检测模拟自然生活状态下大鼠的学习记忆行为。方法：自制微机控制的多功能训练系统,设计训练模型,即指定通过、交替选择和择洞逃避。结果：三个新模型能较好地检测大鼠日常生活有关学习记忆行为。结论：模型具备客观、形象直观和省时的特点     

The learning skills of primates: The rhesus macaque in comparative perspective

Twenty-month-old rhesus monkeys were tested in a modified discrimination-reversal paradigm, which was designed to distinguish abstract learning from stimulus-response associational learning. Previous studies indicate that talapoin monkeys learn associationally and great apes via forming abstract concepts. Adult rhesus monkeys are apparently capable of forming simple abstractions, but learn primarily through associational process. The results of this study show the adolescent rhesus monkeys to be associational learners, with their response patterns indicating more complexity than the talapoins but less than the adult rhesus monkeys. The data suggest that rhesus monkeys develop their low-level capacity of abstract learning with maturation.     

Structure Learning in a Sensorimotor Association Task

Learning is often understood as an organism''s gradual acquisition of the association between a given sensory stimulus and the correct motor response. Mathematically, this corresponds to regressing a mapping between the set of observations and the set of actions. Recently, however, it has been shown both in cognitive and motor neuroscience that humans are not only able to learn particular stimulus-response mappings, but are also able to extract abstract structural invariants that facilitate generalization to novel tasks. Here we show how such structure learning can enhance facilitation in a sensorimotor association task performed by human subjects. Using regression and reinforcement learning models we show that the observed facilitation cannot be explained by these basic models of learning stimulus-response associations. We show, however, that the observed data can be explained by a hierarchical Bayesian model that performs structure learning. In line with previous results from cognitive tasks, this suggests that hierarchical Bayesian inference might provide a common framework to explain both the learning of specific stimulus-response associations and the learning of abstract structures that are shared by different task environments.     

Fragment-based learning of visual object categories in non-human primates

When we perceive a visual object, we implicitly or explicitly associate it with an object category we know. Recent research has shown that the visual system can use local, informative image fragments of a given object, rather than the whole object, to classify it into a familiar category. We have previously reported, using human psychophysical studies, that when subjects learn new object categories using whole objects, they incidentally learn informative fragments, even when not required to do so. However, the neuronal mechanisms by which we acquire and use informative fragments, as well as category knowledge itself, have remained unclear. Here we describe the methods by which we adapted the relevant human psychophysical methods to awake, behaving monkeys and replicated key previous psychophysical results. This establishes awake, behaving monkeys as a useful system for future neurophysiological studies not only of informative fragments in particular, but also of object categorization and category learning in general.     

Thermal and EMG Biofeedback Learning in Nonhuman Primates

Four monkeys were found able to learn to raise and lower hand temperature and to reduce muscle tension to low levels using feedback from the target physiological system. The establishment of this model of biofeedback learning in monkeys enables work on mechanisms mediating the modes of biofeedback most used in clinical practice. Results suggest that biofeedback learning does not need to be mediated by the type of human-specific cognitive strategies employed by humans.     

Monkeys monitor human goals in a nonmatch-to-goal interactive task

We designed a new task, called nonmatch-to-goal, to study the ability of macaque monkeys to interact with humans in a rule-guided paradigm. In this task the monkeys were required to choose one of two targets, from a list of three. For each choice, they were required to switch from their choice on the previous trial to a different one. In a subset of trials the monkeys observed a human partner performing the task. When the human concluded his turn, the monkeys were required to switch to a new goal discarding the human's previous goal. We found that monkeys were very skillful in monitoring goals, not only of their own choice by also those of their human partner. They showed also a surprising ability to coordinate their actions, taking turns with the human partner, starting and stopping their own turn following the decision of the human partner in the task.     

Easy Rider: Monkeys Learn to Drive a Wheelchair to Navigate through a Complex Maze

The neurological bases of spatial navigation are mainly investigated in rodents and seldom in primates. The few studies led on spatial navigation in both human and non-human primates are performed in virtual, not in real environments. This is mostly because of methodological difficulties inherent in conducting research on freely-moving monkeys in real world environments. There is some incertitude, however, regarding the extrapolation of rodent spatial navigation strategies to primates. Here we present an entirely new platform for investigating real spatial navigation in rhesus monkeys. We showed that monkeys can learn a pathway by using different strategies. In these experiments three monkeys learned to drive the wheelchair and to follow a specified route through a real maze. After learning the route, probe tests revealed that animals successively use three distinct navigation strategies based on i) the place of the reward, ii) the direction taken to obtain reward or iii) a cue indicating reward location. The strategy used depended of the options proposed and the duration of learning. This study reveals that monkeys, like rodents and humans, switch between different spatial navigation strategies with extended practice, implying well-conserved brain learning systems across different species. This new task with freely driving monkeys provides a good support for the electrophysiological and pharmacological investigation of spatial navigation in the real world by making possible electrophysiological and pharmacological investigations.     

An observational learning task using Barnes maze in rats

Observational learning, which modulates one’s own behavior by observing the adaptive behavior of others, is crucial for behaving efficiently in social communities. Although many behavioral experiments have reported observational learning in monkeys and humans, its neural mechanisms are still unknown. In order to conduct neuroscientific researches with recording neural activities, we developed an observational learning task for rats. We designed the task using Barnes circular maze and then tested whether rats (observers) could actually improve their learning by observing the behavior of other rats (models) that had already acquired the task. The result showed that the observer rats, which were located in a metal wire mesh cylinder at the center of the maze and allowed to observe model rats escaping to the goal in the maze, demonstrated significantly faster escape behavior than the model rats. Thus, the present study confirmed that rats can efficiently learn the behavioral task by observing the behavior of other rats; this shows that it is conceivable to elucidate the neural mechanisms of social interaction by analyzing neural activity in observer rats performing the observational learning task.     

Social facilitation of exploratory foraging behavior in capuchin monkeys (Cebus apella)

Much of the research on animal social learning focuses on complex cognitive functions such as imitation and emulation. When compelling evidence for such processes is not forthcoming, simpler processes are often assumed but rarely directly tested for. In this study we address the phenomenon of social facilitation, whereby the presence of a feeding conspecific is hypothesized to affect the motivation and behavior of the subject, elevating the likelihood of exploration and discovery in relation to the task at hand. Using a novel foraging task, sufficiently challenging that only just over half the subjects successfully gained food from it, we compared the performance of capuchin monkeys working either alone, or in a “social” condition where an actively feeding conspecific was in an adjacent chamber. Although similar numbers of subjects in these conditions were eventually successful during the 20 trials presented, the latency to successful solution of the task was over three times faster for monkeys in the social condition. The minority of monkeys that failed to learn (9/23) were then exposed to a proficient model. Only those older than 5 years provided evidence of learning from this. Accordingly, we obtained evidence for the social facilitation the study was designed to test for, and limited supplementary evidence for social learning in the older individuals who had not learned individually. These results are discussed in relation to other recent evidence for social learning in monkeys. Am. J. Primatol. 71:419–426, 2009. © 2009 Wiley‐Liss, Inc.     

Social enhancement can create adaptive,arbitrary and maladaptive cultural traditions

Many animals are known to learn socially, i.e. they are able to acquire new behaviours by using information from other individuals. Researchers distinguish between a number of different social-learning mechanisms such as imitation and social enhancement. Social enhancement is a simple form of social learning that is among the most widespread in animals. However, unlike imitation, it is debated whether social enhancement can create cultural traditions. Based on a recent study on capuchin monkeys, we developed an agent-based model to test the hypotheses that (i) social enhancement can create and maintain stable traditions and (ii) social enhancement can create cultural conformity. Our results supported both hypotheses. A key factor that led to the creation of cultural conformity and traditions was the repeated interaction of individual reinforcement and social enhancement learning. This result emphasizes that the emergence of cultural conformity does not necessarily require cognitively complex mechanisms such as ‘copying the majority’ or group norms. In addition, we observed that social enhancement can create learning dynamics similar to a ‘copy when uncertain’ learning strategy. Results from additional analyses also point to situations that should favour the evolution of learning mechanisms more sophisticated than social enhancement.     

A Bayesian Developmental Approach to Robotic Goal-Based Imitation Learning

A fundamental challenge in robotics today is building robots that can learn new skills by observing humans and imitating human actions. We propose a new Bayesian approach to robotic learning by imitation inspired by the developmental hypothesis that children use self-experience to bootstrap the process of intention recognition and goal-based imitation. Our approach allows an autonomous agent to: (i) learn probabilistic models of actions through self-discovery and experience, (ii) utilize these learned models for inferring the goals of human actions, and (iii) perform goal-based imitation for robotic learning and human-robot collaboration. Such an approach allows a robot to leverage its increasing repertoire of learned behaviors to interpret increasingly complex human actions and use the inferred goals for imitation, even when the robot has very different actuators from humans. We demonstrate our approach using two different scenarios: (i) a simulated robot that learns human-like gaze following behavior, and (ii) a robot that learns to imitate human actions in a tabletop organization task. In both cases, the agent learns a probabilistic model of its own actions, and uses this model for goal inference and goal-based imitation. We also show that the robotic agent can use its probabilistic model to seek human assistance when it recognizes that its inferred actions are too uncertain, risky, or impossible to perform, thereby opening the door to human-robot collaboration.     

A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback

Reward-modulated spike-timing-dependent plasticity (STDP) has recently emerged as a candidate for a learning rule that could explain how behaviorally relevant adaptive changes in complex networks of spiking neurons could be achieved in a self-organizing manner through local synaptic plasticity. However, the capabilities and limitations of this learning rule could so far only be tested through computer simulations. This article provides tools for an analytic treatment of reward-modulated STDP, which allows us to predict under which conditions reward-modulated STDP will achieve a desired learning effect. These analytical results imply that neurons can learn through reward-modulated STDP to classify not only spatial but also temporal firing patterns of presynaptic neurons. They also can learn to respond to specific presynaptic firing patterns with particular spike patterns. Finally, the resulting learning theory predicts that even difficult credit-assignment problems, where it is very hard to tell which synaptic weights should be modified in order to increase the global reward for the system, can be solved in a self-organizing manner through reward-modulated STDP. This yields an explanation for a fundamental experimental result on biofeedback in monkeys by Fetz and Baker. In this experiment monkeys were rewarded for increasing the firing rate of a particular neuron in the cortex and were able to solve this extremely difficult credit assignment problem. Our model for this experiment relies on a combination of reward-modulated STDP with variable spontaneous firing activity. Hence it also provides a possible functional explanation for trial-to-trial variability, which is characteristic for cortical networks of neurons but has no analogue in currently existing artificial computing systems. In addition our model demonstrates that reward-modulated STDP can be applied to all synapses in a large recurrent neural network without endangering the stability of the network dynamics.