Knowledge extraction from evolving spiking neural networks with rank order population coding

This paper demonstrates how knowledge can be extracted from evolving spiking neural networks with rank order population coding. Knowledge discovery is a very important feature of intelligent systems. Yet, a disproportionally small amount of research is centered on the issue of knowledge extraction from spiking neural networks which are considered to be the third generation of artificial neural networks. The lack of knowledge representation compatibility is becoming a major detriment to end users of these networks. We show that a high-level knowledge can be obtained from evolving spiking neural networks. More specifically, we propose a method for fuzzy rule extraction from an evolving spiking network with rank order population coding. The proposed method was used for knowledge discovery on two benchmark taste recognition problems where the knowledge learnt by an evolving spiking neural network was extracted in the form of zero-order Takagi-Sugeno fuzzy IF-THEN rules.     

Rule extraction from minimal neural networks for credit card screening

While feedforward neural networks have been widely accepted as effective tools for solving classification problems, the issue of finding the best network architecture remains unresolved, particularly so in real-world problem settings. We address this issue in the context of credit card screening, where it is important to not only find a neural network with good predictive performance but also one that facilitates a clear explanation of how it produces its predictions. We show that minimal neural networks with as few as one hidden unit provide good predictive accuracy, while having the added advantage of making it easier to generate concise and comprehensible classification rules for the user. To further reduce model size, a novel approach is suggested in which network connections from the input units to this hidden unit are removed by a very straightaway pruning procedure. In terms of predictive accuracy, both the minimized neural networks and the rule sets generated from them are shown to compare favorably with other neural network based classifiers. The rules generated from the minimized neural networks are concise and thus easier to validate in a real-life setting.     

Quarnet Inference Rules for Level-1 Networks

An important problem in phylogenetics is the construction of phylogenetic trees. One way to approach this problem, known as the supertree method, involves inferring a phylogenetic tree with leaves consisting of a set X of species from a collection of trees, each having leaf-set some subset of X. In the 1980s, Colonius and Schulze gave certain inference rules for deciding when a collection of 4-leaved trees, one for each 4-element subset of X, can be simultaneously displayed by a single supertree with leaf-set X. Recently, it has become of interest to extend this and related results to phylogenetic networks. These are a generalization of phylogenetic trees which can be used to represent reticulate evolution (where species can come together to form a new species). It has recently been shown that a certain type of phylogenetic network, called a (unrooted) level-1 network, can essentially be constructed from 4-leaved trees. However, the problem of providing appropriate inference rules for such networks remains unresolved. Here, we show that by considering 4-leaved networks, called quarnets, as opposed to 4-leaved trees, it is possible to provide such rules. In particular, we show that these rules can be used to characterize when a collection of quarnets, one for each 4-element subset of X, can all be simultaneously displayed by a level-1 network with leaf-set X. The rules are an intriguing mixture of tree inference rules, and an inference rule for building up a cyclic ordering of X from orderings on subsets of X of size 4. This opens up several new directions of research for inferring phylogenetic networks from smaller ones, which could yield new algorithms for solving the supernetwork problem in phylogenetics.     

Information storage matrices in neural networks

A new neural network model with feedback based on the concept of information storage matrices is proposed. This model is similar to the Hopfield and spectral type neural networks but has a more general structure. The presentation gives a fully developed theory for first-order networks, including results on the formation of fixed points and their domains of attraction. These results are used to determine, in deterministic sense, the information storage capacity. The algorithm is applied to the DNA sequencing problem. It is demonstrated how a hidden genetic information in an arbitrary long DNA strand can be extracted.     

Neural network models for promoter recognition

The problem of recognition of promoter sites in the DNA sequence has been treated with models of learning neural networks. The maximum network capacity admissible for this problem has been estimated on the basis of the total of experimental data available on the determined promoter sequences. The model of a block neural network has been constructed to satisfy this estimate and rules have been elaborated for its learning and testing. The learning process involves a small (of the order of 10%) part of the total set of promoter sequences. During this procedure the neural network develops a system of distinctive features (key words) to be used as a reference in identifying promoters against the background of random sequences. The learning quality is then tested with the whole set. The efficiency of promoter recognition has been found to amount to 94 to 99%. The probability of an arbitrary sequence being identified as a promoter is 2 to 6%.     

Inferring models of gene expression dynamics

We study the problem of identifying genetic networks in which expression dynamics are modeled by a differential equation that uses logical rules to specify time derivatives. We make three main contributions. First, we describe computationally efficient procedures for identifying the structure and dynamics of such networks from expression time series. Second, we derive predictions for the expected amount of data needed to identify randomly generated networks. Third, if expression values are available for only some of the genes, we show that the structure of the network for these "visible" genes can be identified and that the size and overall complexity of the network can be estimated. We validate these procedures and predictions using simulation experiments based on randomly generated networks with up to 30,000 genes and 17 distinct regulators per gene and on a network that models floral morphogenesis in Arabidopsis thaliana.     

Evolutionary optimization and neural network models of behavior

One of the main challenges to the adaptionist program in general and the use of optimization models in behavioral and evolutionary ecology, in particular, is that organisms are so constrained' by ontogeny and phylogeny that they may not be able to attain optimal solutions, however those are defined. This paper responds to the challenge through the comparison of optimality and neural network models for the behavior of an individual polychaete worm. The evolutionary optimization model is used to compute behaviors (movement in and out of a tube) that maximize a measure of Darwinian fitness based on individual survival and reproduction. The neural network involves motor, sensory, energetic reserve and clock neuronal groups. Ontogeny of the neural network is the change of connections of a single individual in response to its experiences in the environment. Evolution of the neural network is the natural selection of initial values of connections between groups and learning rules for changing connections. Taken together, these can be viewed as design parameters. The best neural networks have fitnesses between 85% and 99% of the fitness of the evolutionary optimization model. More complicated models for polychaete worms are discussed. Formulation of a neural network model for host acceptance decisions by tephritid fruit flies leads to predictions about the neurobiology of the flies. The general conclusion is that neural networks appear to be sufficiently rich and plastic that even weak evolution of design parameters may be sufficient for organisms to achieve behaviors that give fitnesses close to the evolutionary optimal fitness, particularly if the behaviors are relatively simple.     

Neural Network Models for Promoter Recognition

Abstract

The problem of recognition of promoter sites in the DNA sequence has been treated with models of learning neural networks. The maximum network capacity admissible for this problem has been estimated on the basis of the total of experimental data available on the determined promoter sequences. The model of a block neural network has been constructed to satisfy this estimate and rules have been elaborated for its learning and testing. The learning process involves a small (of the order of 10%) part of the total set of promoter sequences. During this procedure the neural network develops a system of distinctive features (key words) to be used as a reference in identifying promoters against the background of random sequences. The learning quality is then tested with the whole set. The efficiency of promoter recognition has been found to amount to 94 to 99%. The probability of an arbitrary sequence being identified as a promoter is 2 to 6%.     

VLSI implementation of neural networks

Currently, fuzzy controllers are the most popular choice for hardware implementation of complex control surfaces because they are easy to design. Neural controllers are more complex and hard to train, but provide an outstanding control surface with much less error than that of a fuzzy controller. There are also some problems that have to be solved before the networks can be implemented on VLSI chips. First, an approximation function needs to be developed because CMOS neural networks have an activation function different than any function used in neural network software. Next, this function has to be used to train the network. Finally, the last problem for VLSI designers is the quantization effect caused by discrete values of the channel length (L) and width (W) of MOS transistor geometries. Two neural networks were designed in 1.5 microm technology. Using adequate approximation functions solved the problem of activation function. With this approach, trained networks were characterized by very small errors. Unfortunately, when the weights were quantized, errors were increased by an order of magnitude. However, even though the errors were enlarged, the results obtained from neural network hardware implementations were superior to the results obtained with fuzzy system approach.     

Extensive feature detection of N-terminal protein sorting signals

MOTIVATION: The prediction of localization sites of various proteins is an important and challenging problem in the field of molecular biology. TargetP, by Emanuelsson et al. (J. Mol. Biol., 300, 1005-1016, 2000) is a neural network based system which is currently the best predictor in the literature for N-terminal sorting signals. One drawback of neural networks, however, is that it is generally difficult to understand and interpret how and why they make such predictions. In this paper, we aim to generate simple and interpretable rules as predictors, and still achieve a practical prediction accuracy. We adopt an approach which consists of an extensive search for simple rules and various attributes which is partially guided by human intuition. RESULTS: We have succeeded in finding rules whose prediction accuracies come close to that of TargetP, while still retaining a very simple and interpretable form. We also discuss and interpret the discovered rules.     

Stability of delayed memristive neural networks with time-varying impulses

This paper addresses the stability problem on the memristive neural networks with time-varying impulses. Based on the memristor theory and neural network theory, the model of the memristor-based neural network is established. Different from the most publications on memristive networks with fixed-time impulse effects, we consider the case of time-varying impulses. Both the destabilizing and stabilizing impulses exist in the model simultaneously. Through controlling the time intervals of the stabilizing and destabilizing impulses, we ensure the effect of the impulses is stabilizing. Several sufficient conditions for the globally exponentially stability of memristive neural networks with time-varying impulses are proposed. The simulation results demonstrate the effectiveness of the theoretical results.     

A neural network methodology of quadratic optimization.

According to the basic optimization principle of artificial neural networks, a novel kind of neural network model for solving the quadratic programming problem is presented. The methodology is based on the Lagrange multiplier theory in optimization and seeks to provide solutions satisfying the necessary conditions of optimality. The equilibrium point of the network satisfies the Kuhn-Tucker condition for the problem. The stability and convergency of the neural network is investigated and the strategy of the neural optimization is discussed. The feasibility of the neural network method is verified with the computation examples. Results of the simulation of the neural network to solve optimum problems are presented to illustrate the computational power of the neural network method.     

Pruning artificial neural networks using neural complexity measures

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.     

Inferring general relations between network characteristics from specific network ensembles

Different network models have been suggested for the topology underlying complex interactions in natural systems. These models are aimed at replicating specific statistical features encountered in real-world networks. However, it is rarely considered to which degree the results obtained for one particular network class can be extrapolated to real-world networks. We address this issue by comparing different classical and more recently developed network models with respect to their ability to generate networks with large structural variability. In particular, we consider the statistical constraints which the respective construction scheme imposes on the generated networks. After having identified the most variable networks, we address the issue of which constraints are common to all network classes and are thus suitable candidates for being generic statistical laws of complex networks. In fact, we find that generic, not model-related dependencies between different network characteristics do exist. This makes it possible to infer global features from local ones using regression models trained on networks with high generalization power. Our results confirm and extend previous findings regarding the synchronization properties of neural networks. Our method seems especially relevant for large networks, which are difficult to map completely, like the neural networks in the brain. The structure of such large networks cannot be fully sampled with the present technology. Our approach provides a method to estimate global properties of under-sampled networks in good approximation. Finally, we demonstrate on three different data sets (C. elegans neuronal network, R. prowazekii metabolic network, and a network of synonyms extracted from Roget's Thesaurus) that real-world networks have statistical relations compatible with those obtained using regression models.     

Identification of phytoplankton from flow cytometry data by using radial basis function neural networks.

We describe here the application of a type of artificial neural network, the Gaussian radial basis function (RBF) network, in the identification of a large number of phytoplankton strains from their 11-dimensional flow cytometric characteristics measured by the European Optical Plankton Analyser instrument. The effect of network parameters on optimization is examined. Optimized RBF networks recognized 34 species of marine and freshwater phytoplankton with 91. 5% success overall. The relative importance of each measured parameter in discriminating these data and the behavior of RBF networks in response to data from "novel" species (species not present in the training data) were analyzed.     

Multimodal functional network connectivity: an EEG-fMRI fusion in network space

EEG and fMRI recordings measure the functional activity of multiple coherent networks distributed in the cerebral cortex. Identifying network interaction from the complementary neuroelectric and hemodynamic signals may help to explain the complex relationships between different brain regions. In this paper, multimodal functional network connectivity (mFNC) is proposed for the fusion of EEG and fMRI in network space. First, functional networks (FNs) are extracted using spatial independent component analysis (ICA) in each modality separately. Then the interactions among FNs in each modality are explored by Granger causality analysis (GCA). Finally, fMRI FNs are matched to EEG FNs in the spatial domain using network-based source imaging (NESOI). Investigations of both synthetic and real data demonstrate that mFNC has the potential to reveal the underlying neural networks of each modality separately and in their combination. With mFNC, comprehensive relationships among FNs might be unveiled for the deep exploration of neural activities and metabolic responses in a specific task or neurological state.     

Learning-Induced Synchronization and Plasticity of a Developing Neural Network

Learning-induced synchronization of a neural network at various developing stages is studied by computer simulations using a pulse-coupled neural network model in which the neuronal activity is simulated by a one-dimensional map. Two types of Hebbian plasticity rules are investigated and their differences are compared. For both models, our simulations show a logarithmic increase in the synchronous firing frequency of the network with the culturing time of the neural network. This result is consistent with recent experimental observations. To investigate how to control the synchronization behavior of a neural network after learning, we compare the occurrence of synchronization for four networks with different designed patterns under the influence of an external signal. The effect of such a signal on the network activity highly depends on the number of connections between neurons. We discuss the synaptic plasticity and enhancement effects for a random network after learning at various developing stages.