An improvement of extreme learning machine for compact single-hidden-layer feedforward neural networks

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing.     

Short-term prediction of chaotic time series by using RBF network with regression weights

We propose a framework for constructing and training a radial basis function (RBF) neural network. The structure of the gaussian functions is modified using a pseudo-gaussian function (PG) in which two scaling parameters sigma are introduced, which eliminates the symmetry restriction and provides the neurons in the hidden layer with greater flexibility with respect to function approximation. We propose a modified PG-BF (pseudo-gaussian basis function) network in which the regression weights are used to replace the constant weights in the output layer. For this purpose, a sequential learning algorithm is presented to adapt the structure of the network, in which it is possible to create a new hidden unit and also to detect and remove inactive units. A salient feature of the network systems is that the method used for calculating the overall output is the weighted average of the output associated with each receptive field. The superior performance of the proposed PG-BF system over the standard RBF are illustrated using the problem of short-term prediction of chaotic time series.     

A Lempel-Ziv complexity-based neural network pruning algorithm

This paper presents a pruning method for artificial neural networks (ANNs) based on the 'Lempel-Ziv complexity' (LZC) measure. We call this method the 'silent pruning algorithm' (SPA). The term 'silent' is used in the sense that SPA prunes ANNs without causing much disturbance during the network training. SPA prunes hidden units during the training process according to their ranks computed from LZC. LZC extracts the number of unique patterns in a time sequence obtained from the output of a hidden unit and a smaller value of LZC indicates higher redundancy of a hidden unit. SPA has a great resemblance to biological brains since it encourages higher complexity during the training process. SPA is similar to, yet different from, existing pruning algorithms. The algorithm has been tested on a number of challenging benchmark problems in machine learning, including cancer, diabetes, heart, card, iris, glass, thyroid, and hepatitis problems. We compared SPA with other pruning algorithms and we found that SPA is better than the 'random deletion algorithm' (RDA) which prunes hidden units randomly. Our experimental results show that SPA can simplify ANNs with good generalization ability.     

Self-organization of spiking neural network that generates autonomous behavior in a real mobile robot

In this paper, we propose self-organization algorithm of spiking neural network (SNN) applicable to autonomous robot for generation of adoptive and goal-directed behavior. First, we formulated a SNN model whose inputs and outputs were analog and the hidden unites are interconnected each other. Next, we implemented it into a miniature mobile robot Khepera. In order to see whether or not a solution(s) for the given task(s) exists with the SNN, the robot was evolved with the genetic algorithm in the environment. The robot acquired the obstacle avoidance and navigation task successfully, exhibiting the presence of the solution. After that, a self-organization algorithm based on a use-dependent synaptic potentiation and depotentiation at synapses of input layer to hidden layer and of hidden layer to output layer was formulated and implemented into the robot. In the environment, the robot incrementally organized the network and the given tasks were successfully performed. The time needed to acquire the desired adoptive and goal-directed behavior using the proposed self-organization method was much less than that with the genetic evolution, approximately one fifth.     

Quantitative analysis of protein far UV circular dichroism spectra by neural networks.

A new method based on neural network theory is presented to analyze and quantify the information content of far UV circular dichroism spectra. Using a backpropagation network model with a single hidden layer between input and output, it was possible to deduce five different secondary structure fractions (helix, parallel and antiparallel beta-sheet, beta-turn and random coil) with satisfactory correlations between calculated and measured secondary structure data. We demonstrate that for each wavelength interval a specific network is suitable. The remaining discrepancy between the secondary structure data from neural network prediction and crystallography may be attributed to errors in the determination of protein concentration and random noise in the CD signal, as indicated by simulations.     

基于BP神经网络与ETM+遥感数据的盐城滨海自然湿地覆被分类

高效而精确的湿地遥感分类是大范围湿地资源动态监测与管理的必要保障。本研究使用ETM 遥感数据,借助Matlab神经网络工具箱,构建了基于BP神经网络的滨海湿地覆被分类模型,并将其应用于江苏盐城沿海湿地珍禽国家级自然保护区的核心区的自然湿地覆被分类研究中。本研究选择3、4、7、8波段作为输入层变量,单隐藏层设为10个节点,输出层变量对应待划分的8种覆被类型,构建三层式BP神经网络滨海湿地覆被分类模型。结果显示,BP分类总精度为85.91%,Kappa系数为0.8328,与最小距离法和极大似然法的分类总精度相比,分别提高了7.99%和6.08%,Kappa系数也相比提高。研究结果表明,BP神经网络分类法是一种较为有效的湿地遥感影像分类技术,能够提高分类精度。     

Quantitative measurement of two-component pH-sensitive colorimetric spectra using multilayer neural networks

The purpose of this research was to develop a noise tolerant and faster processing approach for in vivo and in vitro spectrophotometric applications where distorted spectra are difficult to interpret quantitatively. A PC based multilayer neural network with a sigmoid activation function and a generalized delta learning rule was trained with a two component (protonated and unprotonated form) pH-dependent spectrum generated from microspectrophotometry of the vital dye neutral red (NR). The network makes use of the digitized absorption spectrum between 375 and 675 nm. The number of nodes in the input layer was determined by the required resolution. The number of output nodes determined the step size of the quantization value used to distinguish the input spectra (i.e. defined the number of distinct output steps). Mathematic analysis provided the conditions for which this network is guaranteed to converge. Simulation results showed that features of the input spectrum were successfully identified and stored in the weight matrix of the input and hidden layers. After convergent training with typical spectra, a calibration curve was constructed to interpret the output layer activity and therefore, predict interpolated pH values of unknown spectra. With its built-in redundant presentation, this approach needed no preprocessing procedures (baseline correction or intensive signal averaging) normally used in multicomponent analyses. The identification of unknown spectra with the activities of the output layer is a one step process using the convergent weight matrix. After learning from examples, real time applications can be accomplished without solving multiple linear equations as in the multiple linear regression method. This method can be generalized to pattern oriented sensory information processing and multi-sensor data fusion for quantitative measurement purposes.     

A note on the low-dimensional display of multivariate data using neural networks

A novel neural network technique has been proposed (Livingstone et al. J. Mol. Graphics 1991, 9, 115-118) which is useful for a low-dimensional display of multivariate data sets. The method makes use of the activity values of the hidden neurons in a trained three-layer feed-forward network to produce the low-dimensional display. It was claimed that in contrast to conventional techniques, such as principal components analysis or nonlinear mapping, this technique could be used also to reconstruct, from a given point in the low-dimensional display, the corresponding multivariate input vector via the completely known weight matrices of a suitably trained network. We show here that this claim is unjustified in this general form. When previously unknown, grossly different input vectors are presented to the trained network, they can occupy, for example, exactly the same point in the low-dimensional display which is occupied also by a given training vector, if certain linear relationships between the vector components are fulfilled. Thus, an infinite set of different linearly dependent input vectors is projected onto one single point in the low-dimensional display. Reconstruction of a multivariate vector, starting from this point in the low-dimensional display, is able to lead back to only one multivariate vector (in the example given, to the original training vector).     

Reliable prediction of T-cell epitopes using neural networks with novel sequence representations

In this paper we describe an improved neural network method to predict T-cell class I epitopes. A novel input representation has been developed consisting of a combination of sparse encoding, Blosum encoding, and input derived from hidden Markov models. We demonstrate that the combination of several neural networks derived using different sequence-encoding schemes has a performance superior to neural networks derived using a single sequence-encoding scheme. The new method is shown to have a performance that is substantially higher than that of other methods. By use of mutual information calculations we show that peptides that bind to the HLA A*0204 complex display signal of higher order sequence correlations. Neural networks are ideally suited to integrate such higher order correlations when predicting the binding affinity. It is this feature combined with the use of several neural networks derived from different and novel sequence-encoding schemes and the ability of the neural network to be trained on data consisting of continuous binding affinities that gives the new method an improved performance. The difference in predictive performance between the neural network methods and that of the matrix-driven methods is found to be most significant for peptides that bind strongly to the HLA molecule, confirming that the signal of higher order sequence correlation is most strongly present in high-binding peptides. Finally, we use the method to predict T-cell epitopes for the genome of hepatitis C virus and discuss possible applications of the prediction method to guide the process of rational vaccine design.     

Estimating Selection Coefficients in Spatially Structured Populations from Time Series Data of Allele Frequencies

Inferring the nature and magnitude of selection is an important problem in many biological contexts. Typically when estimating a selection coefficient for an allele, it is assumed that samples are drawn from a panmictic population and that selection acts uniformly across the population. However, these assumptions are rarely satisfied. Natural populations are almost always structured, and selective pressures are likely to act differentially. Inference about selection ought therefore to take account of structure. We do this by considering evolution in a simple lattice model of spatial population structure. We develop a hidden Markov model based maximum-likelihood approach for estimating the selection coefficient in a single population from time series data of allele frequencies. We then develop an approximate extension of this to the structured case to provide a joint estimate of migration rate and spatially varying selection coefficients. We illustrate our method using classical data sets of moth pigmentation morph frequencies, but it has wide applications in settings ranging from ecology to human evolution.     

Convergence analysis of cascade error projection--an efficient learning algorithm for hardware implementation

In this paper, we present a mathematical foundation, including a convergence analysis, for cascading architecture neural network. Our analysis also shows that the convergence of the cascade architecture neural network is assured because it satisfies Liapunov criteria, in an added hidden unit domain rather than in the time domain. From this analysis, a mathematical foundation for the cascade correlation learning algorithm can be found. Furthermore, it becomes apparent that the cascade correlation scheme is a special case from mathematical analysis in which an efficient hardware learning algorithm called Cascade Error Projection(CEP) is proposed. The CEP provides efficient learning in hardware and it is faster to train, because part of the weights are deterministically obtained, and the learning of the remaining weights from the inputs to the hidden unit is performed as a single-layer perceptron learning with previously determined weights kept frozen. In addition, one can start out with zero weight values (rather than random finite weight values) when the learning of each layer is commenced. Further, unlike cascade correlation algorithm (where a pool of candidate hidden units is added), only a single hidden unit is added at a time. Therefore, the simplicity in hardware implementation is also achieved. Finally, 5- to 8-bit parity and chaotic time series prediction problems are investigated; the simulation results demonstrate that 4-bit or more weight quantization is sufficient for learning neural network using CEP. In addition, it is demonstrated that this technique is able to compensate for less bit weight resolution by incorporating additional hidden units. However, generation result may suffer somewhat with lower bit weight quantization.     

A Novel Approach for Lie Detection Based on F-Score and Extreme Learning Machine

A new machine learning method referred to as F-score_ELM was proposed to classify the lying and truth-telling using the electroencephalogram (EEG) signals from 28 guilty and innocent subjects. Thirty-one features were extracted from the probe responses from these subjects. Then, a recently-developed classifier called extreme learning machine (ELM) was combined with F-score, a simple but effective feature selection method, to jointly optimize the number of the hidden nodes of ELM and the feature subset by a grid-searching training procedure. The method was compared to two classification models combining principal component analysis with back-propagation network and support vector machine classifiers. We thoroughly assessed the performance of these classification models including the training and testing time, sensitivity and specificity from the training and testing sets, as well as network size. The experimental results showed that the number of the hidden nodes can be effectively optimized by the proposed method. Also, F-score_ELM obtained the best classification accuracy and required the shortest training and testing time.     

Automatic particle pickup method using a neural network has high accuracy by applying an initial weight derived from eigenimages: a new reference free method for single-particle analysis

The single-particle analysis is a structure-determining method for electron microscope (EM) images which does not require crystal. In this method, the projections are picked up and averaged by the images of similar Euler angles to improve the signal to noise ratio, and then create a 3-D reconstruction. The selection of a large number of particles from the cryo-EM micrographs is a pre-requisite for obtaining a high resolution. To pickup a low-contrast cryo-EM protein image, we have recently found that a three-layer pyramidal-type neural network is successful in detecting such a faint image, which had been difficult to detect by other methods. The connection weights between the input and hidden layers, which work as a matching filter, have revealed that they reflect characters of the particle projections in the training data. The images stored in terms of the connection weights were complex, more similar to the eigenimages which are created by the principal component analysis of the learning images rather than to the averages of the particle projections. When we set the initial learning weights according to the eigenimages in advance, the learning period was able to be shortened to less than half the time of the NN whose initial weights had been set randomly. Further, the pickup accuracy increased from 90 to 98%, and a combination of the matching filters were found to work as an integrated matching filter there. The integrated filters were amazingly similar to averaged projections and can be used directly as references for further two-dimensional averaging. Therefore, this research also presents a brand-new reference-free method for single-particle analysis.     

Gene Expression and Regulation of Higher Plants Under Soil Water Stress

Higher plants not only provide human beings renewable food, building materials and energy, but also play the most important role in keeping a stable environment on earth. Plants differ from animals in many aspects, but the important is that plants are more easily influenced by environment than animals. Plants have a series of fine mechanisms for responding to environmental changes, which has been established during their long-period evolution and artificial domestication. The machinery related to molecular biology is the most important basis. The elucidation of it will extremely and purposefully promote the sustainable utilization of plant resources and make the best use of its current potential under different scales. This molecular mechanism at least includes drought signal recognition (input), signal transduction (many cascade biochemical reactions are involved in this process), signal output, signal responses and phenotype realization, which is a multi-dimension network system and contains many levels of gene expression and regulation. We will focus on the physiological and molecular adaptive machinery of plants under soil water stress and draw a possible blueprint for it. Meanwhile, the issues and perspectives are also discussed. We conclude that biological measures is the basic solution to solving various types of issues in relation to sustainable development and the plant measures is the eventual way.Key Words: Higher plants, soil water stress, gene regulatory network, drought, anti-drought gene resources, signal, ion homeostasis, physiological mechanisms.     

Constructing a fuzzy rule-based system using the ILFN network and Genetic Algorithm.

In this paper, a method for automatic construction of a fuzzy rule-based system from numerical data using the Incremental Learning Fuzzy Neural (ILFN) network and the Genetic Algorithm is presented. The ILFN network was developed for pattern classification applications. The ILFN network, which employed fuzzy sets and neural network theory, equips with a fast, one-pass, on-line, and incremental learning algorithm. After trained, the ILFN network stored numerical knowledge in hidden units, which can then be directly interpreted into if then rule bases. However, the rules extracted from the ILFN network are not in an optimized fuzzy linguistic form. In this paper, a knowledge base for fuzzy expert system is extracted from the hidden units of the ILFN classifier. A genetic algorithm is then invoked, in an iterative manner, to reduce number of rules and select only discriminate features from input patterns needed to provide a fuzzy rule-based system. Three computer simulations using a simulated 2-D 3-class data, the well-known Fisher's Iris data set, and the Wisconsin breast cancer data set were performed. The fuzzy rule-based system derived from the proposed method achieved 100% and 97.33% correct classification on the 75 patterns for training set and 75 patterns for test set, respectively. For the Wisconsin breast cancer data set, using 400 patterns for training and 299 patterns for testing, the derived fuzzy rule-based system achieved 99.5% and 98.33% correct classification on the training set and the test set, respectively.     

Mode dynamics of interacting neural populations by bi-orthogonal spectral decomposition

A system of coupled bistable Hopf oscillators with an external periodic input source was used to model the ability of interacting neural populations to synchronize and desynchronize in response to variations of the input signal. We propose that, in biological systems, the settings of internal and external coupling strengths will affect the behaviour of the system to a greater degree than the input frequency. While input frequency and coupling strength were varied, the spatio-temporal dynamics of the network was examined by the bi-orthogonal decomposition technique. Within this method, effects of variation of input frequency and coupling strength were analyzed in terms of global, spatial and temporal mode entropy and energy, using the spatio-temporal data of the system. We observed a discontinuous evolution of spatio-temporal patterns depending sensitively on both the input frequency and the internal and external coupling strengths of the network. Received: 10 June 1998 / Accepted in revised form: 9 August 1999     

Emergence of network structure due to spike-timing-dependent plasticity in recurrent neuronal networks. I. Input selectivity–strengthening correlated input pathways

Spike-timing-dependent plasticity (STDP) determines the evolution of the synaptic weights according to their pre- and post-synaptic activity, which in turn changes the neuronal activity. In this paper, we extend previous studies of input selectivity induced by (STDP) for single neurons to the biologically interesting case of a neuronal network with fixed recurrent connections and plastic connections from external pools of input neurons. We use a theoretical framework based on the Poisson neuron model to analytically describe the network dynamics (firing rates and spike-time correlations) and thus the evolution of the synaptic weights. This framework incorporates the time course of the post-synaptic potentials and synaptic delays. Our analysis focuses on the asymptotic states of a network stimulated by two homogeneous pools of “steady” inputs, namely Poisson spike trains which have fixed firing rates and spike-time correlations. The (STDP) model extends rate-based learning in that it can implement, at the same time, both a stabilization of the individual neuron firing rates and a slower weight specialization depending on the input spike-time correlations. When one input pathway has stronger within-pool correlations, the resulting synaptic dynamics induced by (STDP) are shown to be similar to those arising in the case of a purely feed-forward network: the weights from the more correlated inputs are potentiated at the expense of the remaining input connections.     

Does a tree-like phylogeny only exist at the tips in the prokaryotes?

The extent to which prokaryotic evolution has been influenced by horizontal gene transfer (HGT) and therefore might be more of a network than a tree is unclear. Here we use supertree methods to ask whether a definitive prokaryotic phylogenetic tree exists and whether it can be confidently inferred using orthologous genes. We analysed an 11-taxon dataset spanning the deepest divisions of prokaryotic relationships, a 10-taxon dataset spanning the relatively recent gamma-proteobacteria and a 61-taxon dataset spanning both, using species for which complete genomes are available. Congruence among gene trees spanning deep relationships is not better than random. By contrast, a strong, almost perfect phylogenetic signal exists in gamma-proteobacterial genes. Deep-level prokaryotic relationships are difficult to infer because of signal erosion, systematic bias, hidden paralogy and/or HGT. Our results do not preclude levels of HGT that would be inconsistent with the notion of a prokaryotic phylogeny. This approach will help decide the extent to which we can say that there is a prokaryotic phylogeny and where in the phylogeny a cohesive genomic signal exists.     

Parametric construction of episode networks from pseudoperiodic time series based on mutual information

Recently, the construction of networks from time series data has gained widespread interest. In this paper, we develop this area further by introducing a network construction procedure for pseudoperiodic time series. We call such networks episode networks, in which an episode corresponds to a temporal interval of a time series, and which defines a node in the network. Our model includes a number of features which distinguish it from current methods. First, the proposed construction procedure is a parametric model which allows it to adapt to the characteristics of the data; the length of an episode being the parameter. As a direct consequence, networks of minimal size containing the maximal information about the time series can be obtained. In this paper, we provide an algorithm to determine the optimal value of this parameter. Second, we employ estimates of mutual information values to define the connectivity structure among the nodes in the network to exploit efficiently the nonlinearities in the time series. Finally, we apply our method to data from electroencephalogram (EEG) experiments and demonstrate that the constructed episode networks capture discriminative information from the underlying time series that may be useful for diagnostic purposes.