Learning and coding of concepts in neural networks

Our environment consists of virtually an infinite number of scenarios in which we have to function. In order to respond properly to an incoming stimulus, the brain has first to analyze it, and to find out the basic familiar elements that are part of it. In other words, by using a library which contains a relatively small number of basic concepts, the brain analyzes the multitude of incoming events. Some of those basic concepts are innate, but many of them must be learned, in order to accommodate for the arbitrary environment around us. A classifying box is defined as the neural network that finds out the familiar concepts that are present in an incoming stimulus. Models for classifying boxes are introduced, and possible mechanisms by which they may establish their libraries of concepts are suggested, and then compared and evaluated by computer simulations.     

Energy coding in neural network with inhibitory neurons

This paper aimed at assessing and comparing the effects of the inhibitory neurons in the neural network on the neural energy distribution, and the network activities in the absence of the inhibitory neurons to understand the nature of neural energy distribution and neural energy coding. Stimulus, synchronous oscillation has significant difference between neural networks with and without inhibitory neurons, and this difference can be quantitatively evaluated by the characteristic energy distribution. In addition, the synchronous oscillation difference of the neural activity can be quantitatively described by change of the energy distribution if the network parameters are gradually adjusted. Compared with traditional method of correlation coefficient analysis, the quantitative indicators based on nervous energy distribution characteristics are more effective in reflecting the dynamic features of the neural network activities. Meanwhile, this neural coding method from a global perspective of neural activity effectively avoids the current defects of neural encoding and decoding theory and enormous difficulties encountered. Our studies have shown that neural energy coding is a new coding theory with high efficiency and great potential.     

Efficient computation via sparse coding in electrosensory neural networks

The electric sense combines spatial aspects of vision and touch with temporal features of audition. Its accessible neural architecture shares similarities with mammalian sensory systems and allows for recordings from successive brain areas to test hypotheses about neural coding. Further, electrosensory stimuli encountered during prey capture, navigation, and communication, can be readily synthesized in the laboratory. These features enable analyses of the neural circuitry that reveal general principles of encoding and decoding, such as segregation of information into separate streams and neural response sparsification. A systems level understanding arises via linkage between cellular differentiation and network architecture, revealed by in vitro and in vivo analyses, while computational modeling reveals how single cell dynamics and connectivity shape the sparsification process.     

Functional model of biological neural networks

A functional model of biological neural networks, called temporal hierarchical probabilistic associative memory (THPAM), is proposed in this paper. THPAM comprises functional models of dendritic trees for encoding inputs to neurons, a first type of neuron for generating spike trains, a second type of neuron for generating graded signals to modulate neurons of the first type, supervised and unsupervised Hebbian learning mechanisms for easy learning and retrieving, an arrangement of dendritic trees for maximizing generalization, hardwiring for rotation-translation-scaling invariance, and feedback connections with different delay durations for neurons to make full use of present and past informations generated by neurons in the same and higher layers. These functional models and their processing operations have many functions of biological neural networks that have not been achieved by other models in the open literature and provide logically coherent answers to many long-standing neuroscientific questions. However, biological justifications of these functional models and their processing operations are required for THPAM to qualify as a macroscopic model (or low-order approximate) of biological neural networks.     

Synchrony measures for biological neural networks

 Synchronous firing of a population of neurons has been observed in many experimental preparations; in addition, various mathematical neural network models have been shown, analytically or numerically, to contain stable synchronous solutions. In order to assess the level of synchrony of a particular network over some time interval, quantitative measures of synchrony are needed. We develop here various synchrony measures which utilize only the spike times of the neurons; these measures are applicable in both experimental situations and in computer models. Using a mathematical model of the CA3 region of the hippocampus, we evaluate these synchrony measures and compare them with pictorial representations of network activity. We illustrate how synchrony is lost and synchrony measures change as heterogeneity amongst cells increases. Theoretical expected values of the synchrony measures for different categories of network solutions are derived and compared with results of simulations. Received: 6 June 1994/Accepted in revised form: 13 January 1995     

Erratum to: Functional model of biological neural networks

[This corrects the article DOI: 10.1007/s11571-010-9110-4.].     

Minimum mean square error estimation of connectivity in biological neural networks

A minimum mean square error (MMSE) estimation scheme is employed to identify the synaptic connectivity in neural networks. This new approach can substantially reduce the amount of data and the computational cost involved in the conventional correlation methods, and is suitable for both nonstationary and stationary neuronal firings. Two algorithms are proposed to estimate the synaptic connectivities recursively, one for nonlinear filtering, the other for linear filtering. In addition, the lower and upper bounds for the MMSE estimator are determined. It is shown that the estimators are consistent in quadratic mean. We also demonstrate that the conventional cross-interval histogram is an asymptotic linear MMSE estimator with an inappropriate initial value. Finally, simulations of both nonlinear and linear (Kalman filter) estimates demonstrate that the true connectivity values are approached asymptotically.     

Knowledge extraction from evolving spiking neural networks with rank order population coding

This paper demonstrates how knowledge can be extracted from evolving spiking neural networks with rank order population coding. Knowledge discovery is a very important feature of intelligent systems. Yet, a disproportionally small amount of research is centered on the issue of knowledge extraction from spiking neural networks which are considered to be the third generation of artificial neural networks. The lack of knowledge representation compatibility is becoming a major detriment to end users of these networks. We show that a high-level knowledge can be obtained from evolving spiking neural networks. More specifically, we propose a method for fuzzy rule extraction from an evolving spiking network with rank order population coding. The proposed method was used for knowledge discovery on two benchmark taste recognition problems where the knowledge learnt by an evolving spiking neural network was extracted in the form of zero-order Takagi-Sugeno fuzzy IF-THEN rules.     

Vision-based force measurement using neural networks for biological cell microinjection

This paper presents a vision-based force measurement method using an artificial neural network model. The proposed model is used for measuring the applied load to a spherical biological cell during micromanipulation process. The devised vision-based method is most useful when force measurement capability is required, but it is very challenging or even infeasible to use a force sensor. Artificial neural networks in conjunction with image processing techniques have been used to estimate the applied load to a cell. A bio-micromanipulation system capable of force measurement has also been established in order to collect the training data required for the proposed neural network model. The geometric characterization of zebrafish embryos membranes has been performed during the penetration of the micropipette prior to piercing. The geometric features are extracted from images using image processing techniques. These features have been used to describe the shape and quantify the deformation of the cell at different indentation depths. The neural network is trained by taking the visual data as the input and the measured corresponding force as the output. Once the neural network is trained with sufficient number of data, it can be used as a precise sensor in bio-micromanipulation setups. However, the proposed neural network model is applicable for indentation of any other spherical elastic object. The results demonstrate the capability of the proposed method. The outcomes of this study could be useful for measuring force in biological cell micromanipulation processes such as injection of the mouse oocyte/embryo.     

Information coding and oscillatory activity in synfire neural networks with and without inhibitory coupling

When a population spike (pulse-packet) propagates through a feedforward network with random excitatory connections, it either evolves to a sustained stable level of synchronous activity or fades away (Diesmann et al. in Nature 402:529-533 1999; Cateau and Fukai Neur Netw 14:675-685 2001). Here I demonstrate that in the presence of noise, the probability of the survival of the pulse-packet (or, equivalently, the firing rate of output neurons) reflects the intensity of the input. Furthermore, inhibitory coupling between layers can result in quasi- periodic alternation between several levels of firing activity. These results are obtained by analyzing the evolution of pulse-packet activity as a Markov chain. For the Markov chain analysis, the output of the chain is a linear mapping of the input into a lower-dimensional space, and the eigenvalues and eigenvectors of the transition matrix determine the dynamics of the evolution. Synchronous propagation of firing activity in successive pools of neurons are simulated in networks of integrate-and-fire and compartmental model neurons, and, consistent with the discrete Markov process, the activation of each pool is observed to be predominantly dependent upon the number of cells that fired in the previous pool. Simulation results agree with the numerical solutions of the Markov model. When inhibitory coupling between layers are included in the Markov model, some eigenvalues become complex numbers, implying oscillatory dynamics. The quasiperiodic dynamics is validated with simulation with leaky integrate-and-fire neurons. The networks demonstrate different modes of quasiperiodic activity as the inhibition or excitation parameters of the network are varied.     

Determination of eukaryotic protein coding regions using neural networks and information theory.

Our previous work applied neural network techniques to the problem of discriminating open reading frame (ORF) sequences taken from introns versus exons. The method counted the codon frequencies in an ORF of a specified length, and then used this codon frequency representation of DNA fragments to train a neural net (essentially a Perceptron with a sigmoidal, or "soft step function", output) to perform this discrimination. After training, the network was then applied to a disjoint "predict" set of data to assess accuracy. The resulting accuracy in our previous work was 98.4%, exceeding accuracies reported in the literature at that time for other algorithms. Here, we report even higher accuracies stemming from calculations of mutual information (a correlation measure) of spatially separated codons in exons, and in introns. Significant mutual information exists in exons, but not in introns, between adjacent codons. This suggests that dicodon frequencies of adjacent codons are important for intron/exon discrimination. We report that accuracies obtained using a neural net trained on the frequency of dicodons is significantly higher at smaller fragment lengths than even our original results using codon frequencies, which were already higher than simple statistical methods that also used codon frequencies. We also report accuracies obtained from including codon and dicodon statistics in all six reading frames, i.e. the three frames on the original and complement strand. Inclusion of six-frame statistics increases the accuracy still further. We also compare these neural net results to a Bayesian statistical prediction method that assumes independent codon frequencies in each position. The performance of the Bayesian scheme is poorer than any of the neural based schemes, however many methods reported in the literature either explicitly, or implicitly, use this method. Specifically, Bayesian prediction schemes based on codon frequencies achieve 90.9% accuracy on 90 codon ORFs, while our best neural net scheme reaches 99.4% accuracy on 60 codon ORFs. "Accuracy" is defined as the average of the exon and intron sensitivities. Achievement of sufficiently high accuracies on short fragment lengths can be useful in providing a computational means of finding coding regions in unannotated DNA sequences such as those arising from the mega-base sequencing efforts of the Human Genome Project. We caution that the high accuracies reported here do not represent a complete solution to the problem of identifying exons in "raw" base sequences. The accuracies are considerably lower from exons of small length, although still higher than accuracies reported in the literature for other methods. Short exon lengths are not uncommon.(ABSTRACT TRUNCATED AT 400 WORDS)     

Degenerate coding in neural systems

When the dimensionality of a neural circuit is substantially larger than the dimensionality of the variable it encodes, many different degenerate network states can produce the same output. In this review I will discuss three different neural systems that are linked by this theme. The pyloric network of the lobster, the song control system of the zebra finch, and the odor encoding system of the locust, while different in design, all contain degeneracies between their internal parameters and the outputs they encode. Indeed, although the dynamics of song generation and odor identification are quite different, computationally, odor recognition can be thought of as running the song generation circuitry backwards. In both of these systems, degeneracy plays a vital role in mapping a sparse neural representation devoid of correlations onto external stimuli (odors or song structure) that are strongly correlated. I argue that degeneracy between input and output states is an inherent feature of many neural systems, which can be exploited as a fault-tolerant method of reliably learning, generating, and discriminating closely related patterns.     

Functional identification of biological neural networks using reservoir adaptation for point processes

The complexity of biological neural networks does not allow to directly relate their biophysical properties to the dynamics of their electrical activity. We present a reservoir computing approach for functionally identifying a biological neural network, i.e. for building an artificial system that is functionally equivalent to the reference biological network. Employing feed-forward and recurrent networks with fading memory, i.e. reservoirs, we propose a point process based learning algorithm to train the internal parameters of the reservoir and the connectivity between the reservoir and the memoryless readout neurons. Specifically, the model is an Echo State Network (ESN) with leaky integrator neurons, whose individual leakage time constants are also adapted. The proposed ESN algorithm learns a predictive model of stimulus-response relations in in vitro and simulated networks, i.e. it models their response dynamics. Receiver Operating Characteristic (ROC) curve analysis indicates that these ESNs can imitate the response signal of a reference biological network. Reservoir adaptation improved the performance of an ESN over readout-only training methods in many cases. This also held for adaptive feed-forward reservoirs, which had no recurrent dynamics. We demonstrate the predictive power of these ESNs on various tasks with cultured and simulated biological neural networks.     

Global rhythmic activities in hippocampal neural fields and neural coding

Global oscillations of the neural field represent some of the most interesting expressions of the hippocampal activity, being related also to learning and memory. To study oscillatory activities of the CA3 field in theta range, a model of this sub-field of Hippocampus has been formulated. The model describes the firing activity of CA3 neuronal populations within the frame of a kinetic theory of neural systems and it has been used for computer simulations. The results show that the propagation of activities induced in the neural field by hippocampal afferents occurs only in narrow time windows confined by inhibitory barrages, whose time-course follows the theta rhythm. Moreover, during each period of a theta wave, the entire CA3 field bears a firing activity with peculiar space-time patterns, a sort of specific imprint, which can induce effects with similar patterns on brain regions driven by the hippocampal formation. The simulation has also demonstrated the ability of medial septum to influence the global activity of the CA3 pyramidal population through the control of the population of inhibitory interneurons. At last, the possible involvement of global population oscillations in neural coding has been discussed.     

Cephalopod neural networks

Cephalopods have arguably the largest and most complex nervous systems amongst the invertebrates; but despite the squid giant axon being one of the best studied nerve cells in neuroscience, and the availability of superb information on the morphology of some cephalopod brains, there is surprisingly little known about the operation of the neural networks that underlie the sophisticated range of behaviour these animals display. This review focuses on a few of the best studied neural networks: the giant fiber system, the chromatophore system, the statocyst system, the visual system and the learning and memory system, with a view to summarizing our current knowledge and stimulating new studies, particularly on the activities of identified central neurons, to provide a more complete understanding of networks within the cephalopod nervous system.