Spiking Neurons in a Hierarchical Self-Organizing Map Model Can Learn to Develop Spatial and Temporal Properties of Entorhinal Grid Cells and Hippocampal Place Cells

Medial entorhinal grid cells and hippocampal place cells provide neural correlates of spatial representation in the brain. A place cell typically fires whenever an animal is present in one or more spatial regions, or places, of an environment. A grid cell typically fires in multiple spatial regions that form a regular hexagonal grid structure extending throughout the environment. Different grid and place cells prefer spatially offset regions, with their firing fields increasing in size along the dorsoventral axes of the medial entorhinal cortex and hippocampus. The spacing between neighboring fields for a grid cell also increases along the dorsoventral axis. This article presents a neural model whose spiking neurons operate in a hierarchy of self-organizing maps, each obeying the same laws. This spiking GridPlaceMap model simulates how grid cells and place cells may develop. It responds to realistic rat navigational trajectories by learning grid cells with hexagonal grid firing fields of multiple spatial scales and place cells with one or more firing fields that match neurophysiological data about these cells and their development in juvenile rats. The place cells represent much larger spaces than the grid cells, which enable them to support navigational behaviors. Both self-organizing maps amplify and learn to categorize the most frequent and energetic co-occurrences of their inputs. The current results build upon a previous rate-based model of grid and place cell learning, and thus illustrate a general method for converting rate-based adaptive neural models, without the loss of any of their analog properties, into models whose cells obey spiking dynamics. New properties of the spiking GridPlaceMap model include the appearance of theta band modulation. The spiking model also opens a path for implementation in brain-emulating nanochips comprised of networks of noisy spiking neurons with multiple-level adaptive weights for controlling autonomous adaptive robots capable of spatial navigation.     

A biologically inspired hierarchical goal directed navigation model

We propose an extended version of our previous goal directed navigation model based on forward planning of trajectories in a network of head direction cells, persistent spiking cells, grid cells, and place cells. In our original work the animat incrementally creates a place cell map by random exploration of a novel environment. After the exploration phase, the animat decides on its next movement direction towards a goal by probing linear look-ahead trajectories in several candidate directions while stationary and picking the one activating place cells representing the goal location. In this work we present several improvements over our previous model. We improve the range of linear look-ahead probes significantly by imposing a hierarchical structure on the place cell map consistent with the experimental findings of differences in the firing field size and spacing of grid cells recorded at different positions along the dorsal to ventral axis of entorhinal cortex. The new model represents the environment at different scales by populations of simulated hippocampal place cells with different firing field sizes. Among other advantages this model allows simultaneous constant duration linear look-ahead probes at different scales while significantly extending each probe range. The extension of the linear look-ahead probe range while keeping its duration constant also limits the degrading effects of noise accumulation in the network. We show the extended model’s performance using an animat in a large open field environment.     

Coordinated learning of grid cell and place cell spatial and temporal properties: multiple scales,attention and oscillations

A neural model proposes how entorhinal grid cells and hippocampal place cells may develop as spatial categories in a hierarchy of self-organizing maps (SOMs). The model responds to realistic rat navigational trajectories by learning both grid cells with hexagonal grid firing fields of multiple spatial scales, and place cells with one or more firing fields, that match neurophysiological data about their development in juvenile rats. Both grid and place cells can develop by detecting, learning and remembering the most frequent and energetic co-occurrences of their inputs. The model''s parsimonious properties include: similar ring attractor mechanisms process linear and angular path integration inputs that drive map learning; the same SOM mechanisms can learn grid cell and place cell receptive fields; and the learning of the dorsoventral organization of multiple spatial scale modules through medial entorhinal cortex to hippocampus (HC) may use mechanisms homologous to those for temporal learning through lateral entorhinal cortex to HC (‘neural relativity’). The model clarifies how top-down HC-to-entorhinal attentional mechanisms may stabilize map learning, simulates how hippocampal inactivation may disrupt grid cells, and explains data about theta, beta and gamma oscillations. The article also compares the three main types of grid cell models in the light of recent data.     

Theta phase precession of grid and place cell firing in open environments

Place and grid cells in the rodent hippocampal formation tend to fire spikes at successively earlier phases relative to the local field potential theta rhythm as the animal runs through the cell''s firing field on a linear track. However, this ‘phase precession’ effect is less well characterized during foraging in two-dimensional open field environments. Here, we mapped runs through the firing fields onto a unit circle to pool data from multiple runs. We asked which of seven behavioural and physiological variables show the best circular–linear correlation with the theta phase of spikes from place cells in hippocampal area CA1 and from grid cells from superficial layers of medial entorhinal cortex. The best correlate was the distance to the firing field peak projected onto the animal''s current running direction. This was significantly stronger than other correlates, such as instantaneous firing rate and time-in-field, but similar in strength to correlates with other measures of distance travelled through the firing field. Phase precession was stronger in place cells than grid cells overall, and robust phase precession was seen in traversals through firing field peripheries (although somewhat less than in traversals through the centre), consistent with phase coding of displacement along the current direction. This type of phase coding, of place field distance ahead of or behind the animal, may be useful for allowing calculation of goal directions during navigation.     

Robotic and neuronal simulation of the hippocampus and rat navigation.

The properties of hippocampal place cells are reviewed, with particular attention to the nature of the internal and external signals that support their firing. A neuronal simulation of the firing of place cells in open-field environments of varying shape is presented. This simulation is coupled with an existing model of how place-cell firing can be used to drive navigation, and is tested by implementation as a miniature mobile robot. The sensors on the robot provide visual, odometric and short-range proximity data, which are combined to estimate the distance of the walls of the enclosure from the robot and the robot''s current heading direction. These inputs drive the hippocampal simulation, in which the robot''s location is represented as the firing of place cells. If a goal location is encountered, learning occurs in connections from the concurrently active place cells to a set of ''goal cells'', which guide subsequent navigation, allowing the robot to return to an unmarked location. The system shows good agreement with actual place-cell firing, and makes predictions regarding the firing of cells in the subiculum, the effect of blocking long-term synaptic changes, and the locus of search of rats after deformation of their environment.     

Maintaining a cognitive map in darkness: the need to fuse boundary knowledge with path integration

Spatial navigation requires the processing of complex, disparate and often ambiguous sensory data. The neurocomputations underpinning this vital ability remain poorly understood. Controversy remains as to whether multimodal sensory information must be combined into a unified representation, consistent with Tolman's "cognitive map", or whether differential activation of independent navigation modules suffice to explain observed navigation behaviour. Here we demonstrate that key neural correlates of spatial navigation in darkness cannot be explained if the path integration system acted independently of boundary (landmark) information. In vivo recordings demonstrate that the rodent head direction (HD) system becomes unstable within three minutes without vision. In contrast, rodents maintain stable place fields and grid fields for over half an hour without vision. Using a simple HD error model, we show analytically that idiothetic path integration (iPI) alone cannot be used to maintain any stable place representation beyond two to three minutes. We then use a measure of place stability based on information theoretic principles to prove that featureless boundaries alone cannot be used to improve localization above chance level. Having shown that neither iPI nor boundaries alone are sufficient, we then address the question of whether their combination is sufficient and - we conjecture - necessary to maintain place stability for prolonged periods without vision. We addressed this question in simulations and robot experiments using a navigation model comprising of a particle filter and boundary map. The model replicates published experimental results on place field and grid field stability without vision, and makes testable predictions including place field splitting and grid field rescaling if the true arena geometry differs from the acquired boundary map. We discuss our findings in light of current theories of animal navigation and neuronal computation, and elaborate on their implications and significance for the design, analysis and interpretation of experiments.     

Predictions derived from modelling the hippocampal role in navigation

 A computational model of the lesion and single unit data from navigation in rats is reviewed. The model uses external (visual) and internal (odometric) information from the environment to drive the firing of simulated hippocampal place cells. Constraints on the functional form of these inputs are drawn from experiments using an environment of modifiable shape. The place cell representation is used to guide navigation via the creation of a representation of goal location via Hebbian modification of synaptic strengths. The model includes consideration of the phase of firing of place cells with respect to the theta rhythm of hippocampal EEG. A series of predictions for behavioural and single-unit data in rats are derived from the input and output representations of the model. Received: 15 July 1999 / Accepted in revised form: 20 March 2000     

Independence of landmark and self-motion-guided navigation: a different role for grid cells

Recent interest in the neural bases of spatial navigation stems from the discovery of neuronal populations with strong, specific spatial signals. The regular firing field arrays of medial entorhinal grid cells suggest that they may provide place cells with distance information extracted from the animal''s self-motion, a notion we critically review by citing new contrary evidence. Next, we question the idea that grid cells provide a rigid distance metric. We also discuss evidence that normal navigation is possible using only landmarks, without self-motion signals. We then propose a model that supposes that information flow in the navigational system changes between light and dark conditions. We assume that the true map-like representation is hippocampal and argue that grid cells have a crucial navigational role only in the dark. In this view, their activity in the light is predominantly shaped by landmarks rather than self-motion information, and so follows place cell activity; in the dark, their activity is determined by self-motion cues and controls place cell activity. A corollary is that place cell activity in the light depends on non-grid cells in ventral medial entorhinal cortex. We conclude that analysing navigational system changes between landmark and no-landmark conditions will reveal key functional properties.     

How Entorhinal Grid Cells May Learn Multiple Spatial Scales from a Dorsoventral Gradient of Cell Response Rates in a Self-organizing Map

Place cells in the hippocampus of higher mammals are critical for spatial navigation. Recent modeling clarifies how this may be achieved by how grid cells in the medial entorhinal cortex (MEC) input to place cells. Grid cells exhibit hexagonal grid firing patterns across space in multiple spatial scales along the MEC dorsoventral axis. Signals from grid cells of multiple scales combine adaptively to activate place cells that represent much larger spaces than grid cells. But how do grid cells learn to fire at multiple positions that form a hexagonal grid, and with spatial scales that increase along the dorsoventral axis? In vitro recordings of medial entorhinal layer II stellate cells have revealed subthreshold membrane potential oscillations (MPOs) whose temporal periods, and time constants of excitatory postsynaptic potentials (EPSPs), both increase along this axis. Slower (faster) subthreshold MPOs and slower (faster) EPSPs correlate with larger (smaller) grid spacings and field widths. A self-organizing map neural model explains how the anatomical gradient of grid spatial scales can be learned by cells that respond more slowly along the gradient to their inputs from stripe cells of multiple scales, which perform linear velocity path integration. The model cells also exhibit MPO frequencies that covary with their response rates. The gradient in intrinsic rhythmicity is thus not compelling evidence for oscillatory interference as a mechanism of grid cell firing. A response rate gradient combined with input stripe cells that have normalized receptive fields can reproduce all known spatial and temporal properties of grid cells along the MEC dorsoventral axis. This spatial gradient mechanism is homologous to a gradient mechanism for temporal learning in the lateral entorhinal cortex and its hippocampal projections. Spatial and temporal representations may hereby arise from homologous mechanisms, thereby embodying a mechanistic “neural relativity” that may clarify how episodic memories are learned.     

Computational models of grid cells

Grid cells are space-modulated neurons with periodic firing fields. In moving animals, the multiple firing fields of an individual grid cell form a triangular pattern tiling the entire space available to the animal. Collectively, grid cells are thought to provide a context-independent metric representation of the local environment. Since the discovery of grid cells in 2005, a number of models have been proposed to explain the formation of spatially repetitive firing patterns as well as the conversion of these signals to place signals one synapse downstream in the hippocampus. The present article reviews the most recent developments in our understanding of how grid patterns are generated, maintained, and transformed, with particular emphasis on second-generation computational models that have emerged during the past 2-3 years in response to criticism and new data.     

Increased size and stability of CA1 and CA3 place fields in HCN1 knockout mice

Hippocampal CA1 and CA3 pyramidal neuron place cells encode the spatial location of an animal through localized firing patterns called "place fields." To explore the mechanisms that control place cell firing and their relationship to spatial memory, we studied mice with enhanced spatial memory resulting from forebrain-specific knockout of the HCN1 hyperpolarization-activated cation channel. HCN1 is strongly expressed in CA1 neurons and in entorhinal cortex grid cells, which provide spatial information to the hippocampus. Both CA1 and CA3 place fields were larger but more stable in the knockout mice, with the effect greater in CA1 than CA3. As HCN1 is only weakly expressed in CA3 place cells, their altered activity likely reflects loss of HCN1 in grid cells. The more pronounced changes in CA1 likely reflect the intrinsic contribution of HCN1. The enhanced place field stability may underlie the effect of HCN1 deletion to facilitate spatial learning and memory.     

Avian navigation: from historical to modern concepts

Studies on avian navigation began at the end of the 19th century with testing various hypotheses, followed by large-scale displacement experiments to assess the capacity of the birds' navigational abilities. In the 1950s, the first theoretical concepts were published. Kramer proposed his ‘Map-and-Compass’ model, assuming that birds establish the direction to a distant goal with the help of an external reference, a compass. The model describes homing as a two-step process, with the first step determining the direction to the goal as a compass course and the second step locating this course with the help of a compass. This model was widely accepted when numerous experiments with clock-shifted pigeons demonstrated the use of the sun compass, and thus a general involvement of compass orientation, in homing. The ‘map’ step is assumed to use local site-specific information, which led to the idea of a ‘grid map’ based on environmental gradients. Kramer's model still forms the basis of our present concept on avian homing, yet route integration with the help of an external reference provides an alternative strategy to determine the home course, and the magnetic compass is a second compass mechanism available to birds. These mechanisms are interrelated by ontogenetic learning processes. A two-step process, with the first step providing the compass course and the second step locating this course with the help of a compass, appears to be a common feature of avian navigation tasks, yet the origin of the compass courses differs between tasks according to their nature, with courses acquired by experience for flights within the home range, courses based on navigational processes for returning home, and courses derived from genetically coded information in first-time migrants. Compass orientation thus forms the backbone of the avian navigational system. Copyright 2003 The Association for the Study of Animal Behaviour. Published by Elsevier Science Ltd. All rights reserved.      

Hippocampal Remapping Is Constrained by Sparseness rather than Capacity

Grid cells in the medial entorhinal cortex encode space with firing fields that are arranged on the nodes of spatial hexagonal lattices. Potential candidates to read out the space information of this grid code and to combine it with other sensory cues are hippocampal place cells. In this paper, we investigate a population of grid cells providing feed-forward input to place cells. The capacity of the underlying synaptic transformation is determined by both spatial acuity and the number of different spatial environments that can be represented. The codes for different environments arise from phase shifts of the periodical entorhinal cortex patterns that induce a global remapping of hippocampal place fields, i.e., a new random assignment of place fields for each environment. If only a single environment is encoded, the grid code can be read out at high acuity with only few place cells. A surplus in place cells can be used to store a space code for more environments via remapping. The number of stored environments can be increased even more efficiently by stronger recurrent inhibition and by partitioning the place cell population such that learning affects only a small fraction of them in each environment. We find that the spatial decoding acuity is much more resilient to multiple remappings than the sparseness of the place code. Since the hippocampal place code is sparse, we thus conclude that the projection from grid cells to the place cells is not using its full capacity to transfer space information. Both populations may encode different aspects of space.     

Grid cells, place cells, and geodesic generalization for spatial reinforcement learning

Reinforcement learning (RL) provides an influential characterization of the brain's mechanisms for learning to make advantageous choices. An important problem, though, is how complex tasks can be represented in a way that enables efficient learning. We consider this problem through the lens of spatial navigation, examining how two of the brain's location representations--hippocampal place cells and entorhinal grid cells--are adapted to serve as basis functions for approximating value over space for RL. Although much previous work has focused on these systems' roles in combining upstream sensory cues to track location, revisiting these representations with a focus on how they support this downstream decision function offers complementary insights into their characteristics. Rather than localization, the key problem in learning is generalization between past and present situations, which may not match perfectly. Accordingly, although neural populations collectively offer a precise representation of position, our simulations of navigational tasks verify the suggestion that RL gains efficiency from the more diffuse tuning of individual neurons, which allows learning about rewards to generalize over longer distances given fewer training experiences. However, work on generalization in RL suggests the underlying representation should respect the environment's layout. In particular, although it is often assumed that neurons track location in Euclidean coordinates (that a place cell's activity declines "as the crow flies" away from its peak), the relevant metric for value is geodesic: the distance along a path, around any obstacles. We formalize this intuition and present simulations showing how Euclidean, but not geodesic, representations can interfere with RL by generalizing inappropriately across barriers. Our proposal that place and grid responses should be modulated by geodesic distances suggests novel predictions about how obstacles should affect spatial firing fields, which provides a new viewpoint on data concerning both spatial codes.     

Odor supported place cell model and goal navigation in rodents

Experiments with rodents demonstrate that visual cues play an important role in the control of hippocampal place cells and spatial navigation. Nevertheless, rats may also rely on auditory, olfactory and somatosensory stimuli for orientation. It is also known that rats can track odors or self-generated scent marks to find a food source. Here we model odor supported place cells by using a simple feed-forward network and analyze the impact of olfactory cues on place cell formation and spatial navigation. The obtained place cells are used to solve a goal navigation task by a novel mechanism based on self-marking by odor patches combined with a Q-learning algorithm. We also analyze the impact of place cell remapping on goal directed behavior when switching between two environments. We emphasize the importance of olfactory cues in place cell formation and show that the utility of environmental and self-generated olfactory cues, together with a mixed navigation strategy, improves goal directed navigation.     

Place cells, neocortex and spatial navigation: a short review

Hippocampal place cells are characterized by location-specific firing, that is each cell fires in a restricted region of the environment explored by the rat. In this review, we briefly examine the sensory information used by place cells to anchor their firing fields in space and show that, among the various sensory cues that can influence place cell activity, visual and motion-related cues are the most relevant. We then explore the contribution of several cortical areas to the generation of the place cell signal with an emphasis on the role of the visual cortex and parietal cortex. Finally, we address the functional significance of place cell activity and demonstrate the existence of a clear relationship between place cell positional activity and spatial navigation performance. We conclude that place cells, together with head direction cells, provide information useful for spatially guided movements, and thus provide a unique model of how spatial information is encoded in the brain.     

Differences in Visual-Spatial Input May Underlie Different Compression Properties of Firing Fields for Grid Cell Modules in Medial Entorhinal Cortex

Firing fields of grid cells in medial entorhinal cortex show compression or expansion after manipulations of the location of environmental barriers. This compression or expansion could be selective for individual grid cell modules with particular properties of spatial scaling. We present a model for differences in the response of modules to barrier location that arise from different mechanisms for the influence of visual features on the computation of location that drives grid cell firing patterns. These differences could arise from differences in the position of visual features within the visual field. When location was computed from the movement of visual features on the ground plane (optic flow) in the ventral visual field, this resulted in grid cell spatial firing that was not sensitive to barrier location in modules modeled with small spacing between grid cell firing fields. In contrast, when location was computed from static visual features on walls of barriers, i.e. in the more dorsal visual field, this resulted in grid cell spatial firing that compressed or expanded based on the barrier locations in modules modeled with large spacing between grid cell firing fields. This indicates that different grid cell modules might have differential properties for computing location based on visual cues, or the spatial radius of sensitivity to visual cues might differ between modules.     

A feedforward model for the formation of a grid field where spatial information is provided solely from place cells

Grid cells (GCs) in the medial entorhinal cortex (mEC) have the property of having their firing activity spatially tuned to a regular triangular lattice. Several theoretical models for grid field formation have been proposed, but most assume that place cells (PCs) are a product of the grid cell system. There is, however, an alternative possibility that is supported by various strands of experimental data. Here we present a novel model for the emergence of gridlike firing patterns that stands on two key hypotheses: (1) spatial information in GCs is provided from PC activity and (2) grid fields result from a combined synaptic plasticity mechanism involving inhibitory and excitatory neurons mediating the connections between PCs and GCs. Depending on the spatial location, each PC can contribute with excitatory or inhibitory inputs to GC activity. The nature and magnitude of the PC input is a function of the distance to the place field center, which is inferred from rate decoding. A biologically plausible learning rule drives the evolution of the connection strengths from PCs to a GC. In this model, PCs compete for GC activation, and the plasticity rule favors efficient packing of the space representation. This leads to gridlike firing patterns. In a new environment, GCs continuously recruit new PCs to cover the entire space. The model described here makes important predictions and can represent the feedforward connections from hippocampus CA1 to deeper mEC layers.     

Grid cells use HCN1 channels for spatial scaling

Entorhinal grid cells have periodic, hexagonally patterned firing locations that scale up progressively along the dorsal-ventral axis of medial entorhinal cortex. This topographic expansion corresponds with parallel changes in cellular properties dependent on the hyperpolarization-activated cation current (Ih), which is conducted by hyperpolarization-activated cyclic nucleotide-gated (HCN) channels. To test the hypothesis that grid scale is determined by Ih, we recorded grid cells in mice with forebrain-specific knockout of HCN1. We find that, although the dorsal-ventral gradient of the grid pattern was preserved in HCN1 knockout mice, the size and spacing of the grid fields, as well as the period of the accompanying theta modulation, was expanded at all dorsal-ventral levels. There was no change in theta modulation of simultaneously recorded entorhinal interneurons. These observations raise the possibility that, during self-motion-based navigation, Ih contributes to the gain of the transformation from movement signals to spatial firing fields.     

Coexistence of Reward and Unsupervised Learning During the Operant Conditioning of Neural Firing Rates

A fundamental goal of neuroscience is to understand how cognitive processes, such as operant conditioning, are performed by the brain. Typical and well studied examples of operant conditioning, in which the firing rates of individual cortical neurons in monkeys are increased using rewards, provide an opportunity for insight into this. Studies of reward-modulated spike-timing-dependent plasticity (RSTDP), and of other models such as R-max, have reproduced this learning behavior, but they have assumed that no unsupervised learning is present (i.e., no learning occurs without, or independent of, rewards). We show that these models cannot elicit firing rate reinforcement while exhibiting both reward learning and ongoing, stable unsupervised learning. To fix this issue, we propose a new RSTDP model of synaptic plasticity based upon the observed effects that dopamine has on long-term potentiation and depression (LTP and LTD). We show, both analytically and through simulations, that our new model can exhibit unsupervised learning and lead to firing rate reinforcement. This requires that the strengthening of LTP by the reward signal is greater than the strengthening of LTD and that the reinforced neuron exhibits irregular firing. We show the robustness of our findings to spike-timing correlations, to the synaptic weight dependence that is assumed, and to changes in the mean reward. We also consider our model in the differential reinforcement of two nearby neurons. Our model aligns more strongly with experimental studies than previous models and makes testable predictions for future experiments.