Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.     

Three models of song learning: Evidence from behavior

Research on avian song learning has traditionally been based on an instructional model, as exemplified by the sensorimotor model of song development. Several large-scale, species-wide field studies of learned birdsongs have revealed that variation is narrowly restricted to certain aspects of song structure. Other aspects are sufficiently stereotyped and so widely shared by species' members that they qualify as species-specific universals. The limitations on natural song variation are difficult to reconcile with a fully open, instructive model of song learning. An alternative model based on memorization by selection postulates a system of innate neural templates that facilitate the recognition and rapid memorization of conspecific song patterns. Behavioral evidence compatible with this model includes learning preferences, rapid conspecific song learning, and widespread ocurrence of species-specific song universals that are recognized innately but fail to develop in songs of social isolates. A third model combines instruction, in the memorization phase, with selection during song production. An overproduced repertoire of plastic songs previously memorized by instruction is winnowed by selection imposed during social interactions at the time of adult song crystallization. Selection during production is well established as a factor in the song development of several species, in the form of action-based learning. The possible role of selective processes in song memorization merits further neurobiological investigation. © 1997 John Wiley & Sons, Inc. J Neurobiol 33: 501–516, 1997     

Learning set formation in a group of Guinea baboons (Papio papio): Importance of the behavioral variable measured

A recent report of one-trial learning in a group of saddle-back tamarins (Saguinus fuscicollis) conflicted with views of learning set formation based on traditional techniques employing isolated subjects. An experiment is reported in which a group of Guinea baboons (Papio papio) was given a series of discrimination and habit reversal tasks. Both one-trial learning and learning set formation occurred. Analysis of spatial behavior revealed that the group learned in a single trial to discriminate stocked from unstocked zones. Improvement in successes (reinforced “digging”) was progressive but not rapid, indicating learning set formation. It appears that outcomes depend on the behavioral variables chosen.     

Prosimian learning capacities

A survey of the literature on learning by prosimian primates contradicts a traditional view that they are untrainable. They learn simple operants such as bar pressing readily, work well for the usual rewards, and respond in predictable fashion on schedules of partial reinforcement. Two-dimensional visual discriminations involving pattern and brightness also are learned, as are three-dimensional object discriminations. In the area of complex learning, they not only acquire a variety of difficult tasks, but they also reach a final level of performance that is at or near the level of anthropoids. In each of the following areas, at least one experiment is described in which a prosimian species did as well as one or more anthropoid species: object discrimination learning set, discrimination reversal, delayed response, instrumentation, oddity, and relational problems. The anthropoids to which they are most apt to compare favorably are New World monkeys, a group which they resemble physically, especially with respect to degree of manipulativeness, cortical development, and visual acuity.     

Learning multiple routes in homing pigeons

The aerial lifestyle of central-place foraging birds allows wide-ranging movements, raising fundamental questions about their remarkable navigation and memory systems. For example, we know that pigeons (Columba livia), long-standing models for avian navigation, rely on individually distinct routes when homing from familiar sites. But it remains unknown how they cope with the task of learning several routes in parallel. Here, we examined how learning multiple routes influences homing in pigeons. We subjected groups of pigeons to different training protocols, defined by the sequence in which they were repeatedly released from three different sites, either sequentially, in rotation or randomly. We observed that pigeons from all groups successfully developed and applied memories of the different release sites (RSs), irrespective of the training protocol, and that learning several routes in parallel did not impair their capacity to quickly improve their homing efficiency over multiple releases. Our data also indicated that they coped with increasing RS uncertainty by adjusting both their initial behaviour upon release and subsequent homing efficiency. The results of our study broaden our understanding of avian route following and open new possibilities for studying learning and memory in free-flying animals.     

Transfer index and mediational learning in tufted capuchins (Cebus apella)

The transfer index (TI) is a discrimination reversal paradigm that requires the achievement of a given prereversal criterion of accuracy. The mediational learning (ML) paradigm is a modification of the TI procedure that features the presentation of three different reversal conditions designed to assess whether prereversal learning is based on purely associative processes or mediated by the use of a strategy (win-stay/lose-shift). These two paradigms have been used with apes and several Old World monkey species, proving to be effective tools for the comparison of species on the basis of their transfer abilities and the nature of their learning processes. However, among New World monkeys, only the squirrel monkey has been tested. Capuchin (Cebusspp.) adaptability and their mastery in using tools have led to controversial interpretations of their cognitive and learning skills. We evaluated their mode of learning and the transfer of learning using the TI and the ML paradigms. We tested four tufted capuchins (Cebus apella)in a WGTA using a variety of stimulus object pairs. The results show that they possess rather good transfer abilities and one subject showed an associative learning mode. None of the subjects showed evidence of learning mediated by a win-stay/lose-shift strategy.     

Selectivity in social and asocial learning: investigating the prevalence,effect and development of young children's learning preferences

Culture evolution requires both modification and faithful replication of behaviour, thus it is essential to understand how individuals choose between social and asocial learning. In a quasi-experimental design, 3- and 5-year-olds (176), and adults (52) were presented individually with two novel artificial fruits, and told of the apparatus'' relative difficulty (easy versus hard). Participants were asked if they wanted to attempt the task themselves or watch an experimenter attempt it first; and then had their preference either met or violated. A significant proportion of children and adults (74%) chose to learn socially. For children, this request was efficient, as observing a demonstration made them significantly quicker at the task than learning asocially. However, for 5-year-olds, children who selected asocial learning were also found to be highly efficient at the task, showing that by 5 years children are selective in choosing a learning strategy that is effective for them. Adults further evidenced this trend, and also showed selectivity based on task difficulty. This is the first study to examine the rates, performance outcomes and developmental trajectory of preferences in asocial and social learning, ultimately informing our understanding of innovation.     

Budding biology teachers: what have botanical gardens got to offer inquiry learning

Teachers conceptualise inquiry learning in science learning differently. This is particularly evident when teachers are introduced to inquiry pedagogy within a new context. This exploratory study draws on semi-structured interviews conducted with eight pre-service secondary biology teachers following a day visit with university tutors to the Royal Botanical Gardens, Kew. Emerging findings were: first, pre-service biology teachers’ views of inquiry learning range in sophistication from simple notions of ‘learning from doing’ to complex multi-notions such as student generated questions, developing curiosity and encouraging authentic scientific practices. Second, similarly their views of inquiry learning opportunities in botanical gardens ranged from simply places that offered ‘memorable experiences’ to enabling autonomous learning due to the organism diversity and multiple climates. Pre-service teachers categorised as having unsophisticated views of inquiry learning had limited expectations of botanical gardens as productive learning environments. Third, the majority of pre-service teachers were concerned about managing inquiry learning. A tension was identified between how open-ended an inquiry activity could be whilst ensuring student focus. Further, participants were concerned about the practical management of inquiry learning. We discuss implications for teacher educators and botanical garden educators and the requirement for curriculum development and promotion.     

Learning strategies in matching to sample: if-then and configural learning by pigeons

Pigeons learned a matching-to-sample task with a split training-set design in which half of the stimulus displays were untrained and tested following acquisition. Transfer to the untrained displays along with no novel-stimulus transfer indicated that these pigeons learned the task (partially) via if-then rules. Comparisons to other performance measures indicated that they also partially learned the task via configural learning (learning the gestalt of the whole stimulus display). Differences in the FR-sample requirement (1 vs. 20) had no systematic effect on the type of learning or level of learning obtained. Differences from a previous study [Wright, A.A., 1997. Concept learning and learning strategies. Psychol. Sci. 8, 119-123] are discussed, including the effect of displaying the stimuli vertically (traditional display orientation) or horizontally from the floor.     

Investigating the molecular mechanisms of learning and memory using Caenorhabditis elegans

Learning is an essential biological process for survival since it facilitates behavioural plasticity in response to environmental changes. This process is mediated by a wide variety of genes, mostly expressed in the nervous system. Many studies have extensively explored the molecular and cellular mechanisms underlying learning and memory. This review will focus on the advances gained through the study of the nematode Caenorhabditis elegans. C. elegans provides an excellent system to study learning because of its genetic tractability, in addition to its invariant, compact nervous system (~300 neurons) that is well-characterised at the structural level. Importantly, despite its compact nature, the nematode nervous system possesses a high level of conservation with mammalian systems. These features allow the study of genes within specific sensory-, inter- and motor neurons, facilitating the interrogation of signalling pathways that mediate learning via defined neural circuits. This review will detail how learning and memory can be studied in C. elegans through behavioural paradigms that target distinct sensory modalities. We will also summarise recent studies describing mechanisms through which key molecular and cellular pathways are proposed to affect associative and non-associative forms of learning.

     

Temporal learning of predation risk by embryonic amphibians

For prey species that rely on learning to recognize their predators, natural selection should favour individuals able to learn as early as possible. The earliest point at which individuals can gather information about the identity of their potential predators is during the embryonic stage. Indeed, recent experiments have demonstrated that amphibians can learn to recognize predators prior to hatching. Here, we conditioned woodfrog embryos to recognize predatory salamander cues either in the morning or in the evening, and subsequently exposed the two-week-old tadpoles to salamander cues either in the morning or in the evening, and recorded the intensity of their antipredator behaviour. The data indicate that amphibians learn to recognize potential predators while still in the egg, and also learn the temporal component of this information, which they use later in life, to adjust the intensity of their antipredator responses throughout the day.     

Visual Discrimination Learning in the Small Octopus Octopus ocellatus

Octopus ocellatus is a small benthic species of octopus that is easy to rear and spawns large eggs during a short life cycle. These and other features of O. ocellatus may make it an advantageous subject for a broad range of behavioral studies, including those involving various types of learning. However, no type of learning has been studied in O. ocellatus. In a successive visual discrimination task, in which a ‘positive’ or ‘negative’ stimulus (white or black rectangle) was presented to a subject octopus and appropriate rewards or punishments were given to the subject, the rate of ‘correct’ responses (i.e., touches to the positive stimulus or refraining from the negative stimulus) gradually increased. Moreover, ‘observer’ octopuses that observed another octopus performing a visual discrimination task in which reward was also given to the ‘incorrect’ responses (touches to the negative stimulus) showed a higher ratio of incorrect responses in their test sessions. These results, coupled with the physical characteristics of O. ocellatus, indicate that this species is potentially suitable for neurogenetic and neuroembryological studies of learning.     

Predicting novel substrates for enzymes with minimal experimental effort with active learning

Enzymatic substrate promiscuity is more ubiquitous than previously thought, with significant consequences for understanding metabolism and its application to biocatalysis. This realization has given rise to the need for efficient characterization of enzyme promiscuity. Enzyme promiscuity is currently characterized with a limited number of human-selected compounds that may not be representative of the enzyme's versatility. While testing large numbers of compounds may be impractical, computational approaches can exploit existing data to determine the most informative substrates to test next, thereby more thoroughly exploring an enzyme's versatility. To demonstrate this, we used existing studies and tested compounds for four different enzymes, developed support vector machine (SVM) models using these datasets, and selected additional compounds for experiments using an active learning approach. SVMs trained on a chemically diverse set of compounds were discovered to achieve maximum accuracies of ~80% using ~33% fewer compounds than datasets based on all compounds tested in existing studies. Active learning-selected compounds for testing resolved apparent conflicts in the existing training data, while adding diversity to the dataset. The application of these algorithms to wide arrays of metabolic enzymes would result in a library of SVMs that can predict high-probability promiscuous enzymatic reactions and could prove a valuable resource for the design of novel metabolic pathways.     

转变学习方式,在探究活动中体会成功的乐趣

学生的学习方式一般有接受和发现两种,传统的学习方式过分突出和强调接受与掌握,冷落和忽视发现与探究,从而导致了对学生认识过程的极端处理,摧残了学生的学习兴趣和热情,而以培养创新精神和实践能力为主要目的的探究式学习方式则在培养学生多方面能力的同时给予他们以成功的乐趣。     

Machine learning for epigenetics and future medical applications

Understanding epigenetic processes holds immense promise for medical applications. Advances in Machine Learning (ML) are critical to realize this promise. Previous studies used epigenetic data sets associated with the germline transmission of epigenetic transgenerational inheritance of disease and novel ML approaches to predict genome-wide locations of critical epimutations. A combination of Active Learning (ACL) and Imbalanced Class Learning (ICL) was used to address past problems with ML to develop a more efficient feature selection process and address the imbalance problem in all genomic data sets. The power of this novel ML approach and our ability to predict epigenetic phenomena and associated disease is suggested. The current approach requires extensive computation of features over the genome. A promising new approach is to introduce Deep Learning (DL) for the generation and simultaneous computation of novel genomic features tuned to the classification task. This approach can be used with any genomic or biological data set applied to medicine. The application of molecular epigenetic data in advanced machine learning analysis to medicine is the focus of this review.     

Recruiting machine learning methods for molecular simulations of proteins

Abstract

Molecular dynamics (MD) simulations are critical to understanding the movements of proteins in time. Yet, MD simulations are limited due to the availability of high-resolution protein structures, accuracy of the underlying force-field, computational expense, and difficulty in analysing big data-sets. Machine learning algorithms are now routinely used to circumvent many of these limitations and computational biophysicists are continuously making progress in developing novel applications. Here, we discuss some of these methods, varying from traditional dimensionality reduction approaches to more recent abstractions such as transfer learning and reinforcement learning, and how they have been used to deal with the challenges in MD. We conclude with the prospective issues in the application of machine learning methods in MD, to increase accuracy and efficiency of protein dynamics studies in general.     

The evolutionary basis of human social learning

Humans are characterized by an extreme dependence on culturally transmitted information. Such dependence requires the complex integration of social and asocial information to generate effective learning and decision making. Recent formal theory predicts that natural selection should favour adaptive learning strategies, but relevant empirical work is scarce and rarely examines multiple strategies or tasks. We tested nine hypotheses derived from theoretical models, running a series of experiments investigating factors affecting when and how humans use social information, and whether such behaviour is adaptive, across several computer-based tasks. The number of demonstrators, consensus among demonstrators, confidence of subjects, task difficulty, number of sessions, cost of asocial learning, subject performance and demonstrator performance all influenced subjects' use of social information, and did so adaptively. Our analysis provides strong support for the hypothesis that human social learning is regulated by adaptive learning rules.     

Learning outdoors: male lizards show flexible spatial learning under semi-natural conditions

Spatial cognition is predicted to be a fundamental component of fitness in many lizard species, and yet some studies suggest that it is relatively slow and inflexible. However, such claims are based on work conducted using experimental designs or in artificial contexts that may underestimate their cognitive abilities. We used a biologically realistic experimental procedure (using simulated predatory attacks) to study spatial learning and its flexibility in the lizard Eulamprus quoyii in semi-natural outdoor enclosures under similar conditions to those experienced by lizards in the wild. To evaluate the flexibility of spatial learning, we conducted a reversal spatial-learning task in which positive and negative reinforcements of learnt spatial stimuli were switched. Nineteen (32%) male lizards learnt both tasks within 10 days (spatial task mean: 8.16 ± 0.69 (s.e.) and reversal spatial task mean: 10.74 ± 0.98 (s.e.) trials). We demonstrate that E. quoyii are capable of flexible spatial learning and suggest that future studies focus on a range of lizard species which differ in phylogeny and/or ecology, using biologically relevant cognitive tasks, in an effort to bridge the cognitive divide between ecto- and endotherms.