An Efficient Supervised Training Algorithm for Multilayer Spiking Neural Networks

The spiking neural networks (SNNs) are the third generation of neural networks and perform remarkably well in cognitive tasks such as pattern recognition. The spike emitting and information processing mechanisms found in biological cognitive systems motivate the application of the hierarchical structure and temporal encoding mechanism in spiking neural networks, which have exhibited strong computational capability. However, the hierarchical structure and temporal encoding approach require neurons to process information serially in space and time respectively, which reduce the training efficiency significantly. For training the hierarchical SNNs, most existing methods are based on the traditional back-propagation algorithm, inheriting its drawbacks of the gradient diffusion and the sensitivity on parameters. To keep the powerful computation capability of the hierarchical structure and temporal encoding mechanism, but to overcome the low efficiency of the existing algorithms, a new training algorithm, the Normalized Spiking Error Back Propagation (NSEBP) is proposed in this paper. In the feedforward calculation, the output spike times are calculated by solving the quadratic function in the spike response model instead of detecting postsynaptic voltage states at all time points in traditional algorithms. Besides, in the feedback weight modification, the computational error is propagated to previous layers by the presynaptic spike jitter instead of the gradient decent rule, which realizes the layer-wised training. Furthermore, our algorithm investigates the mathematical relation between the weight variation and voltage error change, which makes the normalization in the weight modification applicable. Adopting these strategies, our algorithm outperforms the traditional SNN multi-layer algorithms in terms of learning efficiency and parameter sensitivity, that are also demonstrated by the comprehensive experimental results in this paper.     

Adaptive Neural Network Structure Optimization Algorithm Based on Dynamic Nodes

Large-scale artificial neural networks have many redundant structures, making the network fall into the issue of local optimization and extended training time. Moreover, existing neural network topology optimization algorithms have the disadvantage of many calculations and complex network structure modeling. We propose a Dynamic Node-based neural network Structure optimization algorithm (DNS) to handle these issues. DNS consists of two steps: the generation step and the pruning step. In the generation step, the network generates hidden layers layer by layer until accuracy reaches the threshold. Then, the network uses a pruning algorithm based on Hebb’s rule or Pearson’s correlation for adaptation in the pruning step. In addition, we combine genetic algorithm to optimize DNS (GA-DNS). Experimental results show that compared with traditional neural network topology optimization algorithms, GA-DNS can generate neural networks with higher construction efficiency, lower structure complexity, and higher classification accuracy.     

A cascading structure and training method for multilayer neural networks

A new structure and training method for multilayer neural networks is presented. The proposed method is based on cascade training of subnetworks and optimizing weights layer by layer. The training procedure is completed in two steps. First, a subnetwork, m inputs and n outputs as the style of training samples, is trained using the training samples. Secondly the outputs of the subnetwork is taken as the inputs and the outputs of the training sample as the desired outputs, another subnetwork with n inputs and n outputs is trained. Finally the two trained subnetworks are connected and a trained multilayer neural networks is created. The numerical simulation results based on both linear least squares back-propagation (LSB) and traditional back-propagation (BP) algorithm have demonstrated the efficiency of the proposed method.     

Discovery of predictive rule sets for chlorophyll-a dynamics in the Nakdong River (Korea) by means of the hybrid evolutionary algorithm HEA

This paper presents a hybrid evolutionary algorithm (HEA) to discover complex rule sets predicting the concentration of chlorophyll-a (Chl.a) based on the measured meteorological, hydrological and limnological variables in the hypertrophic Nakdong River. The HEA is designed: (1) to evolve the structure of rule sets by using genetic programming and (2) to optimise the random parameters in the rule sets by means of a genetic algorithm. Time-series of input–output data from 1995 to 1998 without and with time lags up to 7 days were used for training HEA. Independent input–output data for 1994 were used for testing HEA. HEA successfully discovered rule sets for multiple nonlinear relationships between physical, chemical variables and Chl.a, which proved to be predictive for unseen data as well as explanatory. The comparison of results by HEA and previously applied recurrent artificial neural networks to the same data with input–output time lags of 3 days revealed similar good performances of both methods. The sensitivity analysis for the best performing predictive rule set unraveled relationships between seasons, specific input variables and Chl.a which to some degree correspond with known properties of the Nakdong River. The statistics of numerous random runs of the HEA also allowed determining most relevant input variables without a priori knowledge.     

An improvement of extreme learning machine for compact single-hidden-layer feedforward neural networks

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing.     

Neural network programming in bioprocess variable estimation and state prediction.

A neural network program with efficient learning ability for bioprocess variable estimation and state prediction was developed. A 3 layer, feed-forward neural network architecture was used, and the program was written in Quick C ver 2.5 for an IBM compatible computer with a 80486/33 MHz processor. A back propagation training algorithm was used based on learning by pattern and momentum in a combination as used to adjust the connection of weights of the neurons in adjacent layers. The delta rule was applied in a gradient descent search technique to minimize a cost function equal to the mean square difference between the target and the network output. A non-linear, sigmoidal logistic transfer function was used in squashing the weighted sum of the inputs of each neuron to a limited range output. A good neural network prediction model was obtained by training with a sequence of past time course data of a typical bioprocess. The well trained neural network estimated accurately and rapidly the state variables with or without noise even under varying process dynamics.     

A mixed dispatching rule approach in FMS scheduling

Short-term scheduling in flexible manufacturing systems (FMSs) is a difficult problem because of the complexities and dynamic behavior of FMSs. To solve this problem, a dispatching rule approach is widely used. In this approach, however, a single dispatching rule is usually assigned for all machines in a system during a given scheduling interval. In this paper, a mixed dispatching rule which can assign a different dispatching rule for each machine is proposed. A search algorithm which selects an appropriate mixed dispatching rule using predictions based on discrete event simulation is developed for this approach. The search algorithm for the mixed dispatching rule is described in detail. The effectiveness (in meeting performance criteria) of the mixed dispatching rule and the efficiency of the search algorithm relative to exhaustive search (complete enumeration) is demonstrated on an FMS model. The mixed dispatching rule approach performs up to 15.9% better than the conventional approach, and is 4% better on average. The statistical significance of the results is dicussed.     

A neural network learns trajectory of motion from the least action principle

This paper considers the problem of training layered neural networks to generate sequences of states. Aiming at application for situations when an integral characteristic of the process is known rather than the specific sequence of states we put forward a method in which underlying general principle is used as a foundation for the learning procedure. To illustrate the ability of a network to learn a task and to generalize algorithm we consider an example where a network generates sequences of states referred to as trajectories of motion of a particle under an external field. Training is grounded on the employment of the least action principle. In the course of training at restricted sections of the path the network elaborates a recurrent rule for the trajectory generation. The rule proves to be equivalent to the correct equation of motion for the whole trajectory.     

A novel algorithm with differential evolution and coral reef optimization for extreme learning machine training

Extreme learning machine (ELM) is a novel and fast learning method to train single layer feed-forward networks. However due to the demand for larger number of hidden neurons, the prediction speed of ELM is not fast enough. An evolutionary based ELM with differential evolution (DE) has been proposed to reduce the prediction time of original ELM. But it may still get stuck at local optima. In this paper, a novel algorithm hybridizing DE and metaheuristic coral reef optimization (CRO), which is called differential evolution coral reef optimization (DECRO), is proposed to balance the explorative power and exploitive power to reach better performance. The thought and the implement of DECRO algorithm are discussed in this article with detail. DE, CRO and DECRO are applied to ELM training respectively. Experimental results show that DECRO-ELM can reduce the prediction time of original ELM, and obtain better performance for training ELM than both DE and CRO.     

XCSc: a novel approach to clustering with extended classifier system

In this paper, we propose a novel approach to clustering noisy and complex data sets based on the eXtend Classifier Systems (XCS). The proposed approach, termed XCSc, has three main processes: (a) a learning process to evolve the rule population, (b) a rule compacting process to remove redundant rules after the learning process, and (c) a rule merging process to deal with the overlapping rules that commonly occur between the clusters. In the first process, we have modified the clustering mechanisms of the current available XCS and developed a new accelerate learning method to improve the quality of the evolved rule population. In the second process, an effective rule compacting algorithm is utilized. The rule merging process is based on our newly proposed agglomerative hierarchical rule merging algorithm, which comprises the following steps: (i) all the generated rules are modeled by a graph, with each rule representing a node; (ii) the vertices in the graph are merged to form a number of sub-graphs (i.e. rule clusters) under some pre-defined criteria, which generates the final rule set to represent the clusters; (iii) each data is re-checked and assigned to a cluster that it belongs to, guided by the final rule set. In our experiments, we compared the proposed XCSc with CHAMELEON, a benchmark algorithm well known for its excellent performance, on a number of challenging data sets. The results show that the proposed approach outperforms CHAMELEON in the successful rate, and also demonstrates good stability.     

基于主成分分析的随机森林视网膜OCT图像分层算法研究

视网膜是层状结构,临床上可以根据视网膜层厚度改变对一些疾病进行预测和诊断.为了快速且准确地分割出视网膜的不同层带,本论文提出一种基于主成分分析的随机森林视网膜光学相干断层扫描技术(optical coherence tomography,OCT)图像分层算法.该方法使用主成分分析(principal component analysis,PCA)法对随机森林采集到的特征进行重采样,保留重采样后权重大的特征信息维度,从而消除特征维度间的关联性和信息冗余.结果表明,总特征维度在29维的情况下,保留前18维度训练速度提高了23.20%,14维度训练速度提高了42.38%,而对图像分割精度方面影响较小,实验表明该方法有效地提高了算法的效率.     

A machine learning tool for the diagnosis of SARS-CoV-2 infection from hemogram parameters

Monocytes and neutrophils play key roles in the cytokine storm triggered by SARS-CoV-2 infection, which changes their conformation and function. These changes are detectable at the cellular and molecular level and may be different to what is observed in other respiratory infections. Here, we applied machine learning (ML) to develop and validate an algorithm to diagnose COVID-19 using blood parameters. In this retrospective single-center study, 49 hemogram parameters from 12,321 patients with clinical suspicion of COVID-19 and tested by RT-PCR (4239 positive and 8082 negative) were analysed. The dataset was randomly divided into training and validation sets. Blood cell parameters and patient age were used to construct the predictive model with the support vector machine (SVM) tool. The model constructed from the training set (5936 patients) achieved an accuracy for diagnosis of SARS-CoV-2 infection of 0.952 (95% CI: 0.875–0.892). Test sensitivity and specificity was 0.868 and 0.899, respectively, with a positive (PPV) and negative (NPV) predictive value of 0.896 and 0.872, respectively (prevalence 0.50). The validation set model (4964 patients) achieved an accuracy of 0.894 (95% CI: 0.883–0.903). Test sensitivity and specificity was 0.8922 and 0.8951, respectively, with a positive (PPV) and negative (NPV) predictive value of 0.817 and 0.94, respectively (prevalence 0.34). The area under the receiver operating characteristic curve was 0.952 for the algorithm performance. This algorithm may allow to rule out COVID-19 diagnosis with 94% of probability. This represents a great advance for early diagnostic orientation and guiding clinical decisions.     

基于蛋白质二级结构内容的域结构类预测

运用加入竞争层的BP网络,研究了基于蛋白质二级结构内容的域结构类预测问题.在BP网络中嵌入一竞争,层显著提高了网络预测性能.仅使用了一个小的训练集和简单的网络结构,获得了很高的预测精度自支持精度97.62%,jack-knife测试精度97.62%,及平均外推精度90.74%.在建立更完备的域结构类特征向量和更有代表性的训练集的基础上,所述方法将为蛋白质域结构分类领域提供新的分类基准.     

Feature detection with controlled error rates in LC/MS images

The Median M-N rule is a feature detection algorithm to detect peptide signals in Liquid Chromatography/Mass Spectrometry (LC/MS) images. As the procedure does not adequately control the statistical errors, we investigate an extension of the Median M-N rule to compute a statistical bound on the false-positive rate. We then study the false-negative rate and provide insights on the types of signal that can be detected by the M-N rule and the limit of detection. The resulting feature detection algorithm, which we term Quantile M-N rule, can be used in most feature detection algorithms to provide statistical control of the false-positive and false-negative rate.     

Is it a paradox or misinterpretation?

The paradox recently raised by Wang and Yuan (Proteins 2000;38:165-175) in protein structural class prediction is actually a misinterpretation of the data reported in the literature. The Bayes decision rule, which was deemed by Wang and Yuan to be the most powerful method for predicting protein structural classes based on the amino acid composition, and applied by these investigators to derive the upper limit of prediction rate for structural classes, is actually completely the same as the component-coupled algorithm proposed by previous investigators (Chou et al., Proteins 1998;31:97-103). Owing to lack of a complete or near-complete training data set, the upper limit rate thus derived by these investigators might be both invalid and misleading. Clarification of these points will further stimulate investigation of this interesting area.     

Constructing a fuzzy rule-based system using the ILFN network and Genetic Algorithm.

In this paper, a method for automatic construction of a fuzzy rule-based system from numerical data using the Incremental Learning Fuzzy Neural (ILFN) network and the Genetic Algorithm is presented. The ILFN network was developed for pattern classification applications. The ILFN network, which employed fuzzy sets and neural network theory, equips with a fast, one-pass, on-line, and incremental learning algorithm. After trained, the ILFN network stored numerical knowledge in hidden units, which can then be directly interpreted into if then rule bases. However, the rules extracted from the ILFN network are not in an optimized fuzzy linguistic form. In this paper, a knowledge base for fuzzy expert system is extracted from the hidden units of the ILFN classifier. A genetic algorithm is then invoked, in an iterative manner, to reduce number of rules and select only discriminate features from input patterns needed to provide a fuzzy rule-based system. Three computer simulations using a simulated 2-D 3-class data, the well-known Fisher's Iris data set, and the Wisconsin breast cancer data set were performed. The fuzzy rule-based system derived from the proposed method achieved 100% and 97.33% correct classification on the 75 patterns for training set and 75 patterns for test set, respectively. For the Wisconsin breast cancer data set, using 400 patterns for training and 299 patterns for testing, the derived fuzzy rule-based system achieved 99.5% and 98.33% correct classification on the training set and the test set, respectively.     

On the use of multi-objective evolutionary algorithms for the induction of fuzzy classification rule systems

Extracting comprehensible and general classifiers from data in the form of rule systems is an important task in many problem domains. This study investigates the utility of a multi-objective evolutionary algorithm (MOEA) for this task. Multi-objective evolutionary algorithms are capable of finding several trade-off solutions between different objectives in a single run. In the context of the present study, the objectives to be optimised are the complexity of the rule systems, and their fit to the data. Complex rule systems are required to fit the data well. However, overly complex rule systems often generalise poorly on new data. In addition they tend to be incomprehensible. It is, therefore, important to obtain trade-off solutions that achieve the best possible fit to the data with the lowest possible complexity. The rule systems produced by the proposed multi-objective evolutionary algorithm are compared with those produced by several other existing approaches for a number of benchmark datasets. It is shown that the algorithm produces less complex classifiers that perform well on unseen data.     

肿瘤亚型识别研究中智能算法的应用

目的:为解决肿瘤亚型识别过程中易出现的维数灾难和过拟合问题,提出了一种改进的粒子群BP神经网络集成算法。方法:算法采用欧式距离和互信息来初步过滤冗余基因,之后用Relief算法进一步处理,得到候选特征基因集合。采用BP神经网络作为基分类器,将特征基因提取与分类器训练相结合,改进的粒子群对其权值和阈值进行全局搜索优化。结果:当隐含层神经元个数为5时,候选特征基因个数为110时,QPSO/BP算法全局优化和搜索,此时的分类准确率最高。结论:该算法不但提高了肿瘤分型识别的准确率,而且降低了学习的复杂度。     

Short-term prediction of chaotic time series by using RBF network with regression weights

We propose a framework for constructing and training a radial basis function (RBF) neural network. The structure of the gaussian functions is modified using a pseudo-gaussian function (PG) in which two scaling parameters sigma are introduced, which eliminates the symmetry restriction and provides the neurons in the hidden layer with greater flexibility with respect to function approximation. We propose a modified PG-BF (pseudo-gaussian basis function) network in which the regression weights are used to replace the constant weights in the output layer. For this purpose, a sequential learning algorithm is presented to adapt the structure of the network, in which it is possible to create a new hidden unit and also to detect and remove inactive units. A salient feature of the network systems is that the method used for calculating the overall output is the weighted average of the output associated with each receptive field. The superior performance of the proposed PG-BF system over the standard RBF are illustrated using the problem of short-term prediction of chaotic time series.     

Computational model for perception of objects and motions

Perception of objects and motions in the visual scene is one of the basic problems in the visual system. There exist ‘What’ and ‘Where’ pathways in the superior visual cortex, starting from the simple cells in the primary visual cortex. The former is able to perceive objects such as forms, color, and texture, and the latter perceives ‘where’, for example, velocity and direction of spatial movement of objects. This paper explores brain-like computational architectures of visual information processing. We propose a visual perceptual model and computational mechanism for training the perceptual model. The computational model is a three-layer network. The first layer is the input layer which is used to receive the stimuli from natural environments. The second layer is designed for representing the internal neural information. The connections between the first layer and the second layer, called the receptive fields of neurons, are self-adaptively learned based on principle of sparse neural representation. To this end, we introduce Kullback-Leibler divergence as the measure of independence between neural responses and derive the learning algorithm based on minimizing the cost function. The proposed algorithm is applied to train the basis functions, namely receptive fields, which are localized, oriented, and bandpassed. The resultant receptive fields of neurons in the second layer have the characteristics resembling that of simple cells in the primary visual cortex. Based on these basis functions, we further construct the third layer for perception of what and where in the superior visual cortex. The proposed model is able to perceive objects and their motions with a high accuracy and strong robustness against additive noise. Computer simulation results in the final section show the feasibility of the proposed perceptual model and high efficiency of the learning algorithm.