Self-influencing synaptic plasticity: Recurrent changes of synaptic weights can lead to specific functional properties

Recent experimental results suggest that dendritic and back-propagating spikes can influence synaptic plasticity in different ways (Holthoff, 2004; Holthoff et al., 2005). In this study we investigate how these signals could interact at dendrites in space and time leading to changing plasticity properties at local synapse clusters. Similar to a previous study (Saudargiene et al., 2004) we employ a differential Hebbian learning rule to emulate spike-timing dependent plasticity and investigate how the interaction of dendritic and back-propagating spikes, as the post-synaptic signals, could influence plasticity. Specifically, we will show that local synaptic plasticity driven by spatially confined dendritic spikes can lead to the emergence of synaptic clusters with different properties. If one of these clusters can drive the neuron into spiking, plasticity may change and the now arising global influence of a back-propagating spike can lead to a further segregation of the clusters and possibly the dying-off of some of them leading to more functional specificity. These results suggest that through plasticity being a spatial and temporal local process, the computational properties of dendrites or complete neurons can be substantially augmented. Action Editor: Wulfram Gerstner     

Odor supported place cell model and goal navigation in rodents

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation.     

Path-finding in real and simulated rats: assessing the influence of path characteristics on navigation learning

A large body of experimental evidence suggests that the hippocampal place field system is involved in reward based navigation learning in rodents. Reinforcement learning (RL) mechanisms have been used to model this, associating the state space in an RL-algorithm to the place-field map in a rat. The convergence properties of RL-algorithms are affected by the exploration patterns of the learner. Therefore, we first analyzed the path characteristics of freely exploring rats in a test arena. We found that straight path segments with mean length 23 cm up to a maximal length of 80 cm take up a significant proportion of the total paths. Thus, rat paths are biased as compared to random exploration. Next we designed a RL system that reproduces these specific path characteristics. Our model arena is covered by overlapping, probabilistically firing place fields (PF) of realistic size and coverage. Because convergence of RL-algorithms is also influenced by the state space characteristics, different PF-sizes and densities, leading to a different degree of overlap, were also investigated. The model rat learns finding a reward opposite to its starting point. We observed that the combination of biased straight exploration, overlapping coverage and probabilistic firing will strongly impair the convergence of learning. When the degree of randomness in the exploration is increased, convergence improves, but the distribution of straight path segments becomes unrealistic and paths become 'wiggly'. To mend this situation without affecting the path characteristic two additional mechanisms are implemented: a gradual drop of the learned weights (weight decay) and path length limitation, which prevents learning if the reward is not found after some expected time. Both mechanisms limit the memory of the system and thereby counteract effects of getting trapped on a wrong path. When using these strategies individually divergent cases get substantially reduced and for some parameter settings no divergence was found anymore at all. Using weight decay and path length limitation at the same time, convergence is not much improved but instead time to convergence increases as the memory limiting effect is getting too strong. The degree of improvement relies also on the size and degree of overlap (coverage density) in the place field system. The used combination of these two parameters leads to a trade-off between convergence and speed to convergence. Thus, this study suggests that the role of the PF-system in navigation learning cannot be considered independently from the animals' exploration pattern.     

Developing velocity sensitivity in a model neuron by local synaptic plasticity

Sensor neurons, like those in the visual cortex, display specific functional properties, e.g., tuning for the orientation, direction and velocity of a moving stimulus. It is still unclear how these properties arise from the processing of the inputs which converge at a given cell. Specifically, little is known how such properties can develop by ways of synaptic plasticity. In this study we investigate the hypothesis that velocity sensitivity can develop at a neuron from different types of synaptic plasticity at different dendritic sub-structures. Specifically we are implementing spike-timing dependent plasticity at one dendritic branch and conventional long-term potentiation at another branch, both driven by dendritic spikes triggered by moving inputs. In the first part of the study, we show how velocity sensitivity can arise from such a spatially localized difference in the plasticity. In the second part we show how this scenario is augmented by the interaction between dendritic spikes and back-propagating spikes also at different dendritic branches. Recent theoretical (Saudargiene et al. in Neural Comput 16:595–626, 2004) and experimental (Froemke et al. in Nature 434:221–225, 2005) results on spatially localized plasticity suggest that such processes may play a major role in determining how synapses will change depending on their site. The current study suggests that such mechanisms could be used to develop the functional specificities of a neuron.     

Learning to reach by reinforcement learning using a receptive field based function approximation approach with continuous actions

Reinforcement learning methods can be used in robotics applications especially for specific target-oriented problems, for example the reward-based recalibration of goal directed actions. To this end still relatively large and continuous state-action spaces need to be efficiently handled. The goal of this paper is, thus, to develop a novel, rather simple method which uses reinforcement learning with function approximation in conjunction with different reward-strategies for solving such problems. For the testing of our method, we use a four degree-of-freedom reaching problem in 3D-space simulated by a two-joint robot arm system with two DOF each. Function approximation is based on 4D, overlapping kernels (receptive fields) and the state-action space contains about 10,000 of these. Different types of reward structures are being compared, for example, reward-on- touching-only against reward-on-approach. Furthermore, forbidden joint configurations are punished. A continuous action space is used. In spite of a rather large number of states and the continuous action space these reward/punishment strategies allow the system to find a good solution usually within about 20 trials. The efficiency of our method demonstrated in this test scenario suggests that it might be possible to use it on a real robot for problems where mixed rewards can be defined in situations where other types of learning might be difficult. This work was supported by EU-Grant PACO-PLUS.