The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and the processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural networks. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available measure of importance based on Fisher information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.     

Persistent Synapse Loss Induced by Repetitive LTD in Developing Rat Hippocampal Neurons

Synaptic pruning is a physiological event that eliminates excessive or inappropriate synapses to form proper synaptic connections during development of neurons. Appropriate synaptic pruning is required for normal neural development. However, the mechanism of synaptic pruning is not fully understood. Strength of synaptic activity under competitive circumstances is thought to act as a selective force for synaptic pruning. Long-term depression (LTD) is a synaptic plasticity showing persistent decreased synaptic efficacy, which is accompanied by morphological changes of dendritic spines including transient retraction. Repetitive induction of LTD has been shown to cause persistent loss of synapses in mature neurons. Here, we show that multiple, but not single, induction of LTD caused a persistent reduction in the number of dendritic synapses in cultured rat developing hippocampal neurons. When LTD was induced in 14 days in vitro cultures by application of (RS)-3,5-dihydroxyphenylglycine (DHPG), a group I metabotropic glutamate receptor (mGluR) agonist, and repeated three times with a one day interval, there was a significant decrease in the number of dendritic synapses. This effect continued up to at least two weeks after the triple LTD induction. The persistent reduction in synapse number occurred in the proximal dendrites, but not the distal dendrites, and was prevented by simultaneous application of the group I/II mGluR antagonist (S)-a-methyl-4-carboxyphenylglycine (MCPG). In conclusion, we found that repetitive LTD induction in developing neurons elicits synaptic pruning and contributes to activity-dependent regulation of synapse number in rat hippocampal neurons.     

Rules of engagement: factors that regulate activity-dependent synaptic plasticity during neural network development

Overproduction and pruning during development is a phenomenon that can be observed in the number of organisms in a population, the number of cells in many tissue types, and even the number of synapses on individual neurons. The sculpting of synaptic connections in the brain of a developing organism is guided by its personal experience, which on a neural level translates to specific patterns of activity. Activity-dependent plasticity at glutamatergic synapses is an integral part of neuronal network formation and maturation in developing vertebrate and invertebrate brains. As development of the rodent forebrain transitions away from an over-proliferative state, synaptic plasticity undergoes modification. Late developmental changes in synaptic plasticity signal the establishment of a more stable network and relate to pronounced perceptual and cognitive abilities. In large part, activation of glutamate-sensitive N-methyl-d-aspartate (NMDA) receptors regulates synaptic stabilization during development and is a necessary step in memory formation processes that occur in the forebrain. A developmental change in the subunits that compose NMDA receptors coincides with developmental modifications in synaptic plasticity and cognition, and thus much research in this area focuses on NMDA receptor composition. We propose that there are additional, equally important developmental processes that influence synaptic plasticity, including mechanisms that are upstream (factors that influence NMDA receptors) and downstream (intracellular processes regulated by NMDA receptors) from NMDA receptor activation. The goal of this review is to summarize what is known and what is not well understood about developmental changes in functional plasticity at glutamatergic synapses, and in the end, attempt to relate these changes to maturation of neural networks.     

A Three-Threshold Learning Rule Approaches the Maximal Capacity of Recurrent Neural Networks

Understanding the theoretical foundations of how memories are encoded and retrieved in neural populations is a central challenge in neuroscience. A popular theoretical scenario for modeling memory function is the attractor neural network scenario, whose prototype is the Hopfield model. The model simplicity and the locality of the synaptic update rules come at the cost of a poor storage capacity, compared with the capacity achieved with perceptron learning algorithms. Here, by transforming the perceptron learning rule, we present an online learning rule for a recurrent neural network that achieves near-maximal storage capacity without an explicit supervisory error signal, relying only upon locally accessible information. The fully-connected network consists of excitatory binary neurons with plastic recurrent connections and non-plastic inhibitory feedback stabilizing the network dynamics; the memory patterns to be memorized are presented online as strong afferent currents, producing a bimodal distribution for the neuron synaptic inputs. Synapses corresponding to active inputs are modified as a function of the value of the local fields with respect to three thresholds. Above the highest threshold, and below the lowest threshold, no plasticity occurs. In between these two thresholds, potentiation/depression occurs when the local field is above/below an intermediate threshold. We simulated and analyzed a network of binary neurons implementing this rule and measured its storage capacity for different sizes of the basins of attraction. The storage capacity obtained through numerical simulations is shown to be close to the value predicted by analytical calculations. We also measured the dependence of capacity on the strength of external inputs. Finally, we quantified the statistics of the resulting synaptic connectivity matrix, and found that both the fraction of zero weight synapses and the degree of symmetry of the weight matrix increase with the number of stored patterns.     

Experimental analysis and computational modeling of interburst intervals in spontaneous activity of cortical neuronal culture

Rhythmic bursting is the most striking behavior of cultured cortical networks and may start in the second week after plating. In this study, we focus on the intervals between spontaneously occurring bursts, and compare experimentally recorded values with model simulations. In the models, we use standard neurons and synapses, with physiologically plausible parameters taken from literature. All networks had a random recurrent architecture with sparsely connected neurons. The number of neurons varied between 500 and 5,000. We find that network models with homogeneous synaptic strengths produce asynchronous spiking or stable regular bursts. The latter, however, are in a range not seen in recordings. By increasing the synaptic strength in a (randomly chosen) subset of neurons, our simulations show interburst intervals (IBIs) that agree better with in vitro experiments. In this regime, called weakly synchronized, the models produce irregular network bursts, which are initiated by neurons with relatively stronger synapses. In some noise-driven networks, a subthreshold, deterministic, input is applied to neurons with strong synapses, to mimic pacemaker network drive. We show that models with such "intrinsically active neurons" (pacemaker-driven models) tend to generate IBIs that are determined by the frequency of the fastest pacemaker and do not resemble experimental data. Alternatively, noise-driven models yield realistic IBIs. Generally, we found that large-scale noise-driven neuronal network models required synaptic strengths with a bimodal distribution to reproduce the experimentally observed IBI range. Our results imply that the results obtained from small network models cannot simply be extrapolated to models of more realistic size. Synaptic strengths in large-scale neuronal network simulations need readjustment to a bimodal distribution, whereas small networks do not require such changes.     

Structure of peripheral synapses: autonomic ganglia

Final motor neurons in sympathetic and parasympathetic ganglia receive synaptic inputs from preganglionic neurons. Quantitative ultrastructural analyses have shown that the spatial distribution of these synapses is mostly sparse and random. Typically, only about 1%-2% of the neuronal surface is covered with synapses, with the rest of the neuronal surface being closely enclosed by Schwann cell processes. The number of synaptic inputs is correlated with the dendritic complexity of the target neuron, and the total number of synaptic contacts is related to the surface area of the post-synaptic neuron. Overall, most neurons receive fewer than 150 synaptic contacts, with individual preganglionic inputs providing between 10 and 50 synaptic contacts. This variation is probably one determinant of synaptic strength in autonomic ganglia. Many neurons in prevertebral sympathetic ganglia receive additional convergent synaptic inputs from intestinofugal neurons located in the enteric plexuses. The neurons support these additional inputs via larger dendritic arborisations together with a higher overall synaptic density. There is considerable neurochemical heterogeneity in presynaptic boutons. Some synapses apparently lack most of the proteins normally required for fast transmitter release and probably do not take part in conventional ganglionic transmission. Furthermore, most preganglionic boutons in the ganglionic neuropil do not form direct synaptic contacts with any neurons. Nevertheless, these boutons may well contribute to slow transmission processes that need not require conventional synaptic structures.     

Depression-biased reverse plasticity rule is required for stable learning at top-down connections

Top-down synapses are ubiquitous throughout neocortex and play a central role in cognition, yet little is known about their development and specificity. During sensory experience, lower neocortical areas are activated before higher ones, causing top-down synapses to experience a preponderance of post-synaptic activity preceding pre-synaptic activity. This timing pattern is the opposite of that experienced by bottom-up synapses, which suggests that different versions of spike-timing dependent synaptic plasticity (STDP) rules may be required at top-down synapses. We consider a two-layer neural network model and investigate which STDP rules can lead to a distribution of top-down synaptic weights that is stable, diverse and avoids strong loops. We introduce a temporally reversed rule (rSTDP) where top-down synapses are potentiated if post-synaptic activity precedes pre-synaptic activity. Combining analytical work and integrate-and-fire simulations, we show that only depression-biased rSTDP (and not classical STDP) produces stable and diverse top-down weights. The conclusions did not change upon addition of homeostatic mechanisms, multiplicative STDP rules or weak external input to the top neurons. Our prediction for rSTDP at top-down synapses, which are distally located, is supported by recent neurophysiological evidence showing the existence of temporally reversed STDP in synapses that are distal to the post-synaptic cell body.     

An adaptive visual neuronal model implementing competitive, temporally asymmetric Hebbian learning

A novel depth-from-motion vision model based on leaky integrate-and-fire (I&F) neurons incorporates the implications of recent neurophysiological findings into an algorithm for object discovery and depth analysis. Pulse-coupled I&F neurons capture the edges in an optical flow field and the associated time of travel of those edges is encoded as the neuron parameters, mainly the time constant of the membrane potential and synaptic weight. Correlations between spikes and their timing thus code depth in the visual field. Neurons have multiple output synapses connecting to neighbouring neurons with an initial Gaussian weight distribution. A temporally asymmetric learning rule is used to adapt the synaptic weights online, during which competitive behaviour emerges between the different input synapses of a neuron. It is shown that the competition mechanism can further improve the model performance. After training, the weights of synapses sourced from a neuron do not display a Gaussian distribution, having adapted to encode features of the scenes to which they have been exposed.     

Dynamics of recurrent neural networks with delayed unreliable synapses: metastable clustering

The influence of unreliable synapses on the dynamic properties of a neural network is investigated for a homogeneous integrate-and-fire network with delayed inhibitory synapses. Numerical and analytical calculations show that the network relaxes to a state with dynamic clusters of identical size which permanently exchange neurons. We present analytical results for the number of clusters and their distribution of firing times which are determined by the synaptic properties. The number of possible configurations increases exponentially with network size. In addition to states with a maximal number of clusters, metastable ones with a smaller number of clusters survive for an exponentially large time scale. An externally excited cluster survives for some time, too, thus clusters may encode information.     

mGRASP enables mapping mammalian synaptic connectivity with light microscopy

The GFP reconstitution across synaptic partners (GRASP) technique, based on functional complementation between two nonfluorescent GFP fragments, can be used to detect the location of synapses quickly, accurately and with high spatial resolution. The method has been previously applied in the nematode and the fruit fly but requires substantial modification for use in the mammalian brain. We developed mammalian GRASP (mGRASP) by optimizing transmembrane split-GFP carriers for mammalian synapses. Using in silico protein design, we engineered chimeric synaptic mGRASP fragments that were efficiently delivered to synaptic locations and reconstituted GFP fluorescence in vivo. Furthermore, by integrating molecular and cellular approaches with a computational strategy for the three-dimensional reconstruction of neurons, we applied mGRASP to both long-range circuits and local microcircuits in the mouse hippocampus and thalamocortical regions, analyzing synaptic distribution in single neurons and in dendritic compartments.     

LTP promotes a selective long-term stabilization and clustering of dendritic spines

Dendritic spines are the main postsynaptic site of excitatory contacts between neurons in the central nervous system. On cortical neurons, spines undergo a continuous turnover regulated by development and sensory activity. However, the functional implications of this synaptic remodeling for network properties remain currently unknown. Using repetitive confocal imaging on hippocampal organotypic cultures, we find that learning-related patterns of activity that induce long-term potentiation act as a selection mechanism for the stabilization and localization of spines. Through a lasting N-methyl-D-aspartate receptor and protein synthesis–dependent increase in protrusion growth and turnover, induction of plasticity promotes a pruning and replacement of nonactivated spines by new ones together with a selective stabilization of activated synapses. Furthermore, most newly formed spines preferentially grow in close proximity to activated synapses and become functional within 24 h, leading to a clustering of functional synapses. Our results indicate that synaptic remodeling associated with induction of long-term potentiation favors the selection of inputs showing spatiotemporal interactions on a given neuron.     

Long-Term Relationships between Synaptic Tenacity, Synaptic Remodeling, and Network Activity

Synaptic plasticity is widely believed to constitute a key mechanism for modifying functional properties of neuronal networks. This belief implicitly implies, however, that synapses, when not driven to change their characteristics by physiologically relevant stimuli, will maintain these characteristics over time. How tenacious are synapses over behaviorally relevant time scales? To begin to address this question, we developed a system for continuously imaging the structural dynamics of individual synapses over many days, while recording network activity in the same preparations. We found that in spontaneously active networks, distributions of synaptic sizes were generally stable over days. Following individual synapses revealed, however, that the apparently static distributions were actually steady states of synapses exhibiting continual and extensive remodeling. In active networks, large synapses tended to grow smaller, whereas small synapses tended to grow larger, mainly during periods of particularly synchronous activity. Suppression of network activity only mildly affected the magnitude of synaptic remodeling, but dependence on synaptic size was lost, leading to the broadening of synaptic size distributions and increases in mean synaptic size. From the perspective of individual neurons, activity drove changes in the relative sizes of their excitatory inputs, but such changes continued, albeit at lower rates, even when network activity was blocked. Our findings show that activity strongly drives synaptic remodeling, but they also show that significant remodeling occurs spontaneously. Whereas such spontaneous remodeling provides an explanation for “synaptic homeostasis” like processes, it also raises significant questions concerning the reliability of individual synapses as sites for persistently modifying network function.     

Estradiol induces hypothalamic dendritic spines by enhancing glutamate release: a mechanism for organizational sex differences

The naturally occurring sex difference in dendritic spine number on hypothalamic neurons offers a unique opportunity to investigate mechanisms establishing synaptic patterning during perinatal sensitive periods. A major advantage of the rat as a model of sexual differentiation is that treatment of neonatal females with estradiol will permanently induce the male phenotype. During the development of other systems, exuberant innervation is followed by activity-dependent pruning necessary for elimination of spurious synapses. In contrast, we demonstrate that estradiol-induced organization in the hypothalamus involves the induction of new synapses on dendritic spines. Activation of estrogen receptors by estradiol triggers a nongenomic activation of PI3 kinase that results in enhanced glutamate release from presynaptic neurons. Subsequent activation of ionotropic glutamate receptors activates MAP kinases, thereby inducing dendritic spine formation. These results reveal a transneuronal mechanism by which estradiol acts during a sensitive period to establish a profound and lasting sex difference in hypothalamic synaptic patterning.     

Types, numbers and distribution of synapses on the dendritic tree of an identified visual interneuron in the brain of the locust

The descending contralateral movement detector (DCMD), an identified descending interneuron in the brain of the locust Schistocerca gregaria has been investigated by using light and electron microscopy. We describe the fine structure, distribution and numbers of synapses that it receives from another identified brain neuron, the lobular giant movement detector (LGMD), and from unidentified neurons. The DCMD dendrites emerging from the integrative segment vary in form and number between individuals and sexes but always form a flattened dendritic domain. The arborizations and the integrative segment appear to be exclusively postsynaptic. Two types of synaptic contacts (Type 1 and 2) onto the DCMD can be discerned as having either round (Type 1) or pleiomorphic synaptic vesicles (Type 2) and by large (Type 1) or small (Type 2) subsynaptic appositions. Contact zones of Type 1 synapses are smaller than those of Type 2. LGMD-synapses are of Type 1 and occur intermingled with presynaptic sites of unidentified units. Some branches of the DCMD receiving input from unidentified units are devoid of contacting LGMD processes. Synapses of both types are randomly distributed over the DCMD integrative segment and at fibres with similar sizes.Type 1 synapses are much more frequent than Type 2 synapses and their number is negatively correlated with fibre diameter. For a whole DCMD dendritic arborization, a total of 8500 active zones of chemical synapses has been calculated, including a minimum of 2250 LGMD-synapses and about 1000 Type 2 synapses. The DCMD may thus receive a considerable amount of input from as yet unidentified neurons.     

Activity-dependent clustering of functional synaptic inputs on developing hippocampal dendrites

During brain development, before sensory systems become functional, neuronal networks spontaneously generate repetitive bursts of neuronal activity, which are typically synchronized across many neurons. Such activity patterns have been described on the level of networks and cells, but the fine-structure of inputs received by an individual neuron during spontaneous network activity has not been studied. Here, we used calcium imaging to record activity at many synapses of hippocampal pyramidal neurons simultaneously to establish the activity patterns in the majority of synapses of an entire cell. Analysis of the spatiotemporal patterns of synaptic activity revealed a fine-scale connectivity rule: neighboring synapses (<16?μm intersynapse distance) are more likely to be coactive than synapses that are farther away from each other. Blocking spiking activity or NMDA receptor activation revealed that the clustering of synaptic inputs required neuronal activity, demonstrating a role of developmentally expressed spontaneous activity for connecting neurons with subcellular precision.     

Prenatal testosterone masculinizes synaptic input to gonadotropin-releasing hormone neurons in sheep.

In sheep, the control of tonic and surge GnRH secretion is sexually differentiated by testosterone in utero. However, GnRH neurons are not sexually dimorphic with respect to number, distribution, or gross morphology. Therefore, this study tested the hypothesis that prenatal steroids influence synaptic input to GnRH neurons. We compared the number of synapses on GnRH neurons from male, female, and androgenized female lambs (n = 5 each). Androgenized females were exposed to testosterone during mid-gestation. Yearling lambs were perfused, and GnRH neurons were visualized using the LR-1 antibody. Five to seven GnRH neurons from the rostral preoptic area in each animal were viewed at the ultrastructural level. Afferent synapses and glial ensheathment on each neuron were counted in a single section through the plane of the nucleus. GnRH neurons from females received approximately twice as many contacts (3.6 +/- 0.7 synapses/100 microm plasma membrane) as those from male lambs (1.6 +/- 0.3; p < 0.05), similar to previous reports in rats. In addition, the number of synapses on GnRH neurons from androgenized female lambs (1.5 +/- 0.5) was similar to that from male lambs, suggesting that prenatal steroids give rise to sex differences in synaptic input to GnRH neurons.     

The Influence of Synaptic Weight Distribution on Neuronal Population Dynamics

The manner in which different distributions of synaptic weights onto cortical neurons shape their spiking activity remains open. To characterize a homogeneous neuronal population, we use the master equation for generalized leaky integrate-and-fire neurons with shot-noise synapses. We develop fast semi-analytic numerical methods to solve this equation for either current or conductance synapses, with and without synaptic depression. We show that its solutions match simulations of equivalent neuronal networks better than those of the Fokker-Planck equation and we compute bounds on the network response to non-instantaneous synapses. We apply these methods to study different synaptic weight distributions in feed-forward networks. We characterize the synaptic amplitude distributions using a set of measures, called tail weight numbers, designed to quantify the preponderance of very strong synapses. Even if synaptic amplitude distributions are equated for both the total current and average synaptic weight, distributions with sparse but strong synapses produce higher responses for small inputs, leading to a larger operating range. Furthermore, despite their small number, such synapses enable the network to respond faster and with more stability in the face of external fluctuations.     

Involvement of neuropeptide mechanisms in the integration of heterotopic dentate fascia grafts and recipient brain

Embryonic dentate fascia was grafted into the somatosensory neocortex of adult rats. Nine months post-grafting, the ultrastructural and morphometric analysis of the giant synapses established between the grafted granular neurons and inappropriate targets in the recipient brain was performed. As compared to the intact synaptic endings in the control hippocampus, differences were found in both the number and distribution of large dense-core synaptic vesicles, which store the neuropeptide co-transmitters. The peptidergic vesicle proportion (of total vesicle pool) within the ectopic giant synapses was 5.8 +/- 0.6% (versus 3.3 +/- 0.6% in the control). Clusters of large dense-core vesicles near the active zones of aberrant connections were observed almost 7.9 times more frequently than that of normal contacts. These data provide evidence that neuropeptide transmitters are critical for the maintenance of synaptic connections between the heterotopic dentate grafts and host brain.     

Synergistic NGF/B27 gradients position synapses heterogeneously in 3D micropatterned neural cultures

Native functional brain circuits show different numbers of synapses (synaptic densities) in the cerebral cortex. Until now, different synaptic densities could not be studied in vitro using current cell culture methods for primary neurons. Herein, we present a novel microfluidic based cell culture method that combines 3D micropatterning of hydrogel layers with linear chemical gradient formation. Micropatterned hydrogels were used to encapsulate dissociated cortical neurons in laminar cell layers and neurotrophic factors NGF and B27 were added to influence the formation of synapses. Neurotrophic gradients allowed for the positioning of distinguishable synaptic densities throughout a 3D micropatterned neural culture. NGF and B27 gradients were maintained in the microfluidic device for over two weeks without perfusion pumps by utilizing a refilling procedure. Spatial distribution of synapses was examined with a pre-synaptic marker to determine synaptic densities. From our experiments, we observed that (1) cortical neurons responded only to synergistic NGF/B27 gradients, (2) synaptic density increased proportionally to synergistic NGF/B27 gradients; (3) homogeneous distribution of B27 disturbed cortical neurons in sensing NGF gradients and (4) the cell layer position significantly impacted spatial distribution of synapses.     

Activity-dependent wiring of the developing hippocampal neuronal circuit

In the developing hippocampus, functional excitatory synaptic connections seem to be recruited from a preformed, initially silent synaptic network. This functional synapse induction requires presynaptic action potentials paired with postsynaptic depolarization, thus obeying Hebb's rule of association. During early postnatal development the hippocampus exhibits an endogenous form of patterned neuronal activity that is driven by GABAergic depolarization. We propose that this recurrent activity promotes the input-specific induction of functional synapses in the CA1 region. Thus, activity-dependent synaptic reorganization in the developing hippocampus appears to be dominated by an active recruitment of new synapses rather than an active elimination of redundant connections.