Pruning artificial neural networks using neural complexity measures

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.     

Persistent Synapse Loss Induced by Repetitive LTD in Developing Rat Hippocampal Neurons

Synaptic pruning is a physiological event that eliminates excessive or inappropriate synapses to form proper synaptic connections during development of neurons. Appropriate synaptic pruning is required for normal neural development. However, the mechanism of synaptic pruning is not fully understood. Strength of synaptic activity under competitive circumstances is thought to act as a selective force for synaptic pruning. Long-term depression (LTD) is a synaptic plasticity showing persistent decreased synaptic efficacy, which is accompanied by morphological changes of dendritic spines including transient retraction. Repetitive induction of LTD has been shown to cause persistent loss of synapses in mature neurons. Here, we show that multiple, but not single, induction of LTD caused a persistent reduction in the number of dendritic synapses in cultured rat developing hippocampal neurons. When LTD was induced in 14 days in vitro cultures by application of (RS)-3,5-dihydroxyphenylglycine (DHPG), a group I metabotropic glutamate receptor (mGluR) agonist, and repeated three times with a one day interval, there was a significant decrease in the number of dendritic synapses. This effect continued up to at least two weeks after the triple LTD induction. The persistent reduction in synapse number occurred in the proximal dendrites, but not the distal dendrites, and was prevented by simultaneous application of the group I/II mGluR antagonist (S)-a-methyl-4-carboxyphenylglycine (MCPG). In conclusion, we found that repetitive LTD induction in developing neurons elicits synaptic pruning and contributes to activity-dependent regulation of synapse number in rat hippocampal neurons.     

The molecular signals that regulate activity-dependent synapse refinement in the brain

The formation of appropriate synaptic connections is critical for the proper functioning of the brain. Early in development, neurons form a surplus of immature synapses. To establish efficient, functional neural networks, neurons selectively stabilize active synapses and eliminate less active ones. This process is known as activity-dependent synapse refinement. Defects in this process have been implicated in neuropsychiatric disorders such as schizophrenia and autism. Here we review the manner and mechanisms by which synapse elimination is regulated through activity-dependent competition. We propose a theoretical framework for the molecular mechanisms of synapse refinement, in which three types of signals regulate the refinement. We then describe the identity of these signals and discuss how multiple molecular signals interact to achieve appropriate synapse refinement in the brain.     

Dynamics of pruning in simulated large-scale spiking neural networks

Massive synaptic pruning following over-growth is a general feature of mammalian brain maturation. This article studies the synaptic pruning that occurs in large networks of simulated spiking neurons in the absence of specific input patterns of activity. The evolution of connections between neurons were governed by an original bioinspired spike-timing-dependent synaptic plasticity (STDP) modification rule which included a slow decay term. The network reached a steady state with a bimodal distribution of the synaptic weights that were either incremented to the maximum value or decremented to the lowest value. After 1x10(6) time steps the final number of synapses that remained active was below 10% of the number of initially active synapses independently of network size. The synaptic modification rule did not introduce spurious biases in the geometrical distribution of the remaining active projections. The results show that, under certain conditions, the model is capable of generating spontaneously emergent cell assemblies.     

Decreasing-Rate Pruning Optimizes the Construction of Efficient and Robust Distributed Networks

Robust, efficient, and low-cost networks are advantageous in both biological and engineered systems. During neural network development in the brain, synapses are massively over-produced and then pruned-back over time. This strategy is not commonly used when designing engineered networks, since adding connections that will soon be removed is considered wasteful. Here, we show that for large distributed routing networks, network function is markedly enhanced by hyper-connectivity followed by aggressive pruning and that the global rate of pruning, a developmental parameter not previously studied by experimentalists, plays a critical role in optimizing network structure. We first used high-throughput image analysis techniques to quantify the rate of pruning in the mammalian neocortex across a broad developmental time window and found that the rate is decreasing over time. Based on these results, we analyzed a model of computational routing networks and show using both theoretical analysis and simulations that decreasing rates lead to more robust and efficient networks compared to other rates. We also present an application of this strategy to improve the distributed design of airline networks. Thus, inspiration from neural network formation suggests effective ways to design distributed networks across several domains.     

Autophosphorylation of alphaCaMKII is required for ocular dominance plasticity

Experience is a powerful sculptor of developing neural connections. In the primary visual cortex (V1), cortical connections are particularly susceptible to the effects of sensory manipulation during a postnatal critical period. At the molecular level, this activity-dependent plasticity requires the transformation of synaptic depolarization into changes in synaptic weight. The molecule alpha calcium-calmodulin kinase type II (alphaCaMKII) is known to play a central role in this transformation. Importantly, alphaCaMKII function is modulated by autophosphorylation, which promotes Ca(2+)-independent kinase activity. Here we show that mice possessing a mutant form of alphaCaMKII that is unable to autophosphorylate show impairments in ocular dominance plasticity. These results confirm the importance of alphaCaMKII in visual cortical plasticity and suggest that synaptic changes induced by monocular deprivation are stored specifically in glutamatergic synapses made onto excitatory neurons.     

Rules of engagement: factors that regulate activity-dependent synaptic plasticity during neural network development

Overproduction and pruning during development is a phenomenon that can be observed in the number of organisms in a population, the number of cells in many tissue types, and even the number of synapses on individual neurons. The sculpting of synaptic connections in the brain of a developing organism is guided by its personal experience, which on a neural level translates to specific patterns of activity. Activity-dependent plasticity at glutamatergic synapses is an integral part of neuronal network formation and maturation in developing vertebrate and invertebrate brains. As development of the rodent forebrain transitions away from an over-proliferative state, synaptic plasticity undergoes modification. Late developmental changes in synaptic plasticity signal the establishment of a more stable network and relate to pronounced perceptual and cognitive abilities. In large part, activation of glutamate-sensitive N-methyl-d-aspartate (NMDA) receptors regulates synaptic stabilization during development and is a necessary step in memory formation processes that occur in the forebrain. A developmental change in the subunits that compose NMDA receptors coincides with developmental modifications in synaptic plasticity and cognition, and thus much research in this area focuses on NMDA receptor composition. We propose that there are additional, equally important developmental processes that influence synaptic plasticity, including mechanisms that are upstream (factors that influence NMDA receptors) and downstream (intracellular processes regulated by NMDA receptors) from NMDA receptor activation. The goal of this review is to summarize what is known and what is not well understood about developmental changes in functional plasticity at glutamatergic synapses, and in the end, attempt to relate these changes to maturation of neural networks.     

Flowers and weeds: cell-type specific pruning in the developing visual thalamus

In the first weeks of vertebrate postnatal life, neural networks in the visual thalamus undergo activity-dependent refinement thought to be important for the development of functional vision. This process involves pruning of synaptic connections between retinal ganglion cells and excitatory thalamic neurons that relay signals on to visual areas of the cortex. A recent report in Neural Development shows that this does not occur in inhibitory neurons, questioning our current understanding of the development of mature neural circuits. See research article: http://www.neuraldevelopment.com/content/8/1/24     

Variability v.s. synchronicity of neuronal activity in local cortical network models with different wiring topologies

Dynamical behavior of a biological neuronal network depends significantly on the spatial pattern of synaptic connections among neurons. While neuronal network dynamics has extensively been studied with simple wiring patterns, such as all-to-all or random synaptic connections, not much is known about the activity of networks with more complicated wiring topologies. Here, we examined how different wiring topologies may influence the response properties of neuronal networks, paying attention to irregular spike firing, which is known as a characteristic of in vivo cortical neurons, and spike synchronicity. We constructed a recurrent network model of realistic neurons and systematically rewired the recurrent synapses to change the network topology, from a localized regular and a “small-world” network topology to a distributed random network topology. Regular and small-world wiring patterns greatly increased the irregularity or the coefficient of variation (Cv) of output spike trains, whereas such an increase was small in random connectivity patterns. For given strength of recurrent synapses, the firing irregularity exhibited monotonous decreases from the regular to the random network topology. By contrast, the spike coherence between an arbitrary neuron pair exhibited a non-monotonous dependence on the topological wiring pattern. More precisely, the wiring pattern to maximize the spike coherence varied with the strength of recurrent synapses. In a certain range of the synaptic strength, the spike coherence was maximal in the small-world network topology, and the long-range connections introduced in this wiring changed the dependence of spike synchrony on the synaptic strength moderately. However, the effects of this network topology were not really special in other properties of network activity. Action Editor: Xiao-Jing Wang     

Roles of neurotrophins on synaptic development and functions in the central nervous system

Evidence is emerging to suggest that in addition to their "classical" neurotrophic involvement in the regulation of the differentiation, maturation and survival of neurons, neurotrophins play crucial roles in neural transmission and succeeding activity-dependent plasticity of synapses. Here we discuss: 1) the regulated synthesis and secretion of neurotrophins in response to neural activity, 2) the short- and long-term effects of neurotrophins on neural transmission, and 3) the neurotrophin-induced rearrangement of synaptic networks.     

Astrocyte-dependent circuit remodeling by synapse phagocytosis

In the central nervous system, synaptic pruning, the removal of unnecessary synaptic contacts, is an essential process for proper circuit maturation in neurodevelopment as well as for synaptic homeostasis in the adult stage. Dysregulation of synaptic pruning can contribute to the initiation and progression of various mental disorders, such as schizophrenia and depression, as well as neurodegenerative diseases including Alzheimer's disease. In the past 15 years, pioneering works have demonstrated that different types of glial cells regulate the number of synapses by selectively eliminating them through phagocytic molecular machinery. Although a majority of findings have been focused on microglia, it is increasingly evident that astrocytes function as a critical player in activity-dependent synapse elimination in developing, adult, and diseased brains. In this review, we will discuss recent findings showing the mechanisms and physiological importance of astrocyte-mediated synapse elimination in controlling synapses and circuit homeostasis. We propose that astrocytes play dominant and non-redundant roles in eliminating synapses during the activity-dependent circuit remodeling processes that do not involve neuro-inflammation.     

Emergence of Connectivity Motifs in Networks of Model Neurons with Short- and Long-Term Plastic Synapses

Recent experimental data from the rodent cerebral cortex and olfactory bulb indicate that specific connectivity motifs are correlated with short-term dynamics of excitatory synaptic transmission. It was observed that neurons with short-term facilitating synapses form predominantly reciprocal pairwise connections, while neurons with short-term depressing synapses form predominantly unidirectional pairwise connections. The cause of these structural differences in excitatory synaptic microcircuits is unknown. We show that these connectivity motifs emerge in networks of model neurons, from the interactions between short-term synaptic dynamics (SD) and long-term spike-timing dependent plasticity (STDP). While the impact of STDP on SD was shown in simultaneous neuronal pair recordings in vitro, the mutual interactions between STDP and SD in large networks are still the subject of intense research. Our approach combines an SD phenomenological model with an STDP model that faithfully captures long-term plasticity dependence on both spike times and frequency. As a proof of concept, we first simulate and analyze recurrent networks of spiking neurons with random initial connection efficacies and where synapses are either all short-term facilitating or all depressing. For identical external inputs to the network, and as a direct consequence of internally generated activity, we find that networks with depressing synapses evolve unidirectional connectivity motifs, while networks with facilitating synapses evolve reciprocal connectivity motifs. We then show that the same results hold for heterogeneous networks, including both facilitating and depressing synapses. This does not contradict a recent theory that proposes that motifs are shaped by external inputs, but rather complements it by examining the role of both the external inputs and the internally generated network activity. Our study highlights the conditions under which SD-STDP might explain the correlation between facilitation and reciprocal connectivity motifs, as well as between depression and unidirectional motifs.     

Development of neural circuitry for precise temporal sequences through spontaneous activity, axon remodeling, and synaptic plasticity

Temporally precise sequences of neuronal spikes that span hundreds of milliseconds are observed in many brain areas, including songbird premotor nucleus, cat visual cortex, and primary motor cortex. Synfire chains-networks in which groups of neurons are connected via excitatory synapses into a unidirectional chain-are thought to underlie the generation of such sequences. It is unknown, however, how synfire chains can form in local neural circuits, especially for long chains. Here, we show through computer simulation that long synfire chains can develop through spike-time dependent synaptic plasticity and axon remodeling-the pruning of prolific weak connections that follows the emergence of a finite number of strong connections. The formation process begins with a random network. A subset of neurons, called training neurons, intermittently receive superthreshold external input. Gradually, a synfire chain emerges through a recruiting process, in which neurons within the network connect to the tail of the chain started by the training neurons. The model is robust to varying parameters, as well as natural events like neuronal turnover and massive lesions. Our model suggests that long synfire chain can form during the development through self-organization, and axon remodeling, ubiquitous in developing neural circuits, is essential in the process.     

Plasticity of GABAergic synapses in the neonatal rat hippocampus

While the development and plasticity of excitatory synaptic connections have been studied into detail, little is known about the development of inhibitory synapses. As proposed for excitatory synapses, recent studies have indicated that activity-dependent forms of synaptic plasticity, such as long-term potentiation and long-term depression, may play a role in the establishment of functional inhibitory synaptic connections. Here, I review these different forms of plasticity and focus on their possible role in the developing neuronal network.     

On the role of anatomy in learning by the cerebellar cortex

The properties due to the location of neurons, synapses, and possibly even synaptic channels, in neuron networks are still unknown. Our preliminary results suggest that not only the interconnections but also the relative positions of the different elements in the network are of importance in the learning process in the cerebellar cortex. We have used neural field equations to investigate the mechanisms of learning in the hierarchical neural network. The numerical resolution of these equations reveals two important properties: (i) The hierarchical structure of this network has the expected effect on learning because the flow of information at the neuronal level is controlled by the heterosynaptic effect through the synaptic density-connectivity function, i.e. the action potential field variable is controlled by the synaptic efficacy field variable at different points of the neuron. (ii) The geometry of the system involves different velocities of propagation along different fibers, i.e. different delays between cells, and thus has a stabilizing effect on the dynamics, allowing the Purkinje output to reach a given value. The field model proposed should be useful in the study of the spatial properties of hierarchical biological systems.     

Signal propagation in feedforward neuronal networks with unreliable synapses

In this paper, we systematically investigate both the synfire propagation and firing rate propagation in feedforward neuronal network coupled in an all-to-all fashion. In contrast to most earlier work, where only reliable synaptic connections are considered, we mainly examine the effects of unreliable synapses on both types of neural activity propagation in this work. We first study networks composed of purely excitatory neurons. Our results show that both the successful transmission probability and excitatory synaptic strength largely influence the propagation of these two types of neural activities, and better tuning of these synaptic parameters makes the considered network support stable signal propagation. It is also found that noise has significant but different impacts on these two types of propagation. The additive Gaussian white noise has the tendency to reduce the precision of the synfire activity, whereas noise with appropriate intensity can enhance the performance of firing rate propagation. Further simulations indicate that the propagation dynamics of the considered neuronal network is not simply determined by the average amount of received neurotransmitter for each neuron in a time instant, but also largely influenced by the stochastic effect of neurotransmitter release. Second, we compare our results with those obtained in corresponding feedforward neuronal networks connected with reliable synapses but in a random coupling fashion. We confirm that some differences can be observed in these two different feedforward neuronal network models. Finally, we study the signal propagation in feedforward neuronal networks consisting of both excitatory and inhibitory neurons, and demonstrate that inhibition also plays an important role in signal propagation in the considered networks.     

From Caenorhabditis elegans to the human connectome: a specific modular organization increases metabolic,functional and developmental efficiency

The connectome, or the entire connectivity of a neural system represented by a network, ranges across various scales from synaptic connections between individual neurons to fibre tract connections between brain regions. Although the modularity they commonly show has been extensively studied, it is unclear whether the connection specificity of such networks can already be fully explained by the modularity alone. To answer this question, we study two networks, the neuronal network of Caenorhabditis elegans and the fibre tract network of human brains obtained through diffusion spectrum imaging. We compare them to their respective benchmark networks with varying modularities, which are generated by link swapping to have desired modularity values. We find several network properties that are specific to the neural networks and cannot be fully explained by the modularity alone. First, the clustering coefficient and the characteristic path length of both C. elegans and human connectomes are higher than those of the benchmark networks with similar modularity. High clustering coefficient indicates efficient local information distribution, and high characteristic path length suggests reduced global integration. Second, the total wiring length is smaller than for the alternative configurations with similar modularity. This is due to lower dispersion of connections, which means each neuron in the C. elegans connectome or each region of interest in the human connectome reaches fewer ganglia or cortical areas, respectively. Third, both neural networks show lower algorithmic entropy compared with the alternative arrangements. This implies that fewer genes are needed to encode for the organization of neural systems. While the first two findings show that the neural topologies are efficient in information processing, this suggests that they are also efficient from a developmental point of view. Together, these results show that neural systems are organized in such a way as to yield efficient features beyond those given by their modularity alone.