A global competitive neural network

 A study is presented of a set of coupled nets proposed to function as a global competitive network. One net, of hidden nodes, is composed solely of inhibitory neurons and is excitatorily driven and feeds back in a disinhibitory manner to an input net which itself feeds excitatorily to a (cortical) output net. The manner in which the former input and hidden inhibitory net function so as to enhance outputs as compared with inputs, and the further enhancements when the cortical net is added, are explored both mathematically and by simulation. This is extended to learning on cortical afferent and lateral connections. A global wave structure, arising on the inhibitory net in a similar manner to that of pattern formation in a negative laplacian net, is seen to be important to all of these activities. Simulations are only performed in one dimension, although the global nature of the activity is expected to extend to higher dimensions. Possible implications are briefly discussed. Received: 21 November 1993/Accepted in revised form: 30 June 1994     

Active Semi-Supervised Learning Method with Hybrid Deep Belief Networks

In this paper, we develop a novel semi-supervised learning algorithm called active hybrid deep belief networks (AHD), to address the semi-supervised sentiment classification problem with deep learning. First, we construct the previous several hidden layers using restricted Boltzmann machines (RBM), which can reduce the dimension and abstract the information of the reviews quickly. Second, we construct the following hidden layers using convolutional restricted Boltzmann machines (CRBM), which can abstract the information of reviews effectively. Third, the constructed deep architecture is fine-tuned by gradient-descent based supervised learning with an exponential loss function. Finally, active learning method is combined based on the proposed deep architecture. We did several experiments on five sentiment classification datasets, and show that AHD is competitive with previous semi-supervised learning algorithm. Experiments are also conducted to verify the effectiveness of our proposed method with different number of labeled reviews and unlabeled reviews respectively.     

Prediction of MHC Class I Binding Peptides by a Query Learning Algorithm Based on Hidden Markov Models

A query learning algorithm based on hidden Markov models (HMMs) isdeveloped to design experiments for string analysis and prediction of MHCclass I binding peptides. Query learning is introduced to aim at reducingthe number of peptide binding data for training of HMMs. A multiple numberof HMMs, which will collectively serve as a committee, are trained withbinding data and used for prediction in real-number values. The universeof peptides is randomly sampled and subjected to judgement by the HMMs.Peptides whose prediction is least consistent among committee HMMs aretested by experiment. By iterating the feedback cycle of computationalanalysis and experiment the most wanted information is effectivelyextracted. After 7 rounds of active learning with 181 peptides in all,predictive performance of the algorithm surpassed the so far bestperforming matrix based prediction. Moreover, by combining the bothmethods binder peptides (log Kd < -6) could be predicted with84% accuracy. Parameter distribution of the HMMs that can be inspectedvisually after training further offers a glimpse of dynamic specificity ofthe MHC molecules.     

A high-order graph generating self-organizing structure

A large class of neural network models have their units organized in a lattice with fixed topology or generate their topology during the learning process. These network models can be used as neighborhood preserving map of the input manifold, but such a structure is difficult to manage since these maps are graphs with a number of nodes that is just one or two orders of magnitude less than the number of input points (i.e., the complexity of the map is comparable with the complexity of the manifold) and some hierarchical algorithms were proposed in order to obtain a high-level abstraction of these structures. In this paper a general structure capable to extract high order information from the graph generated by a large class of self-organizing networks is presented. This algorithm will allow to build a two layers hierarchical structure starting from the results obtained by using the suitable neural network for the distribution of the input data. Moreover the proposed algorithm is also capable to build a topology preserving map if it is trained using a graph that is also a topology preserving map.     

Informational properties of neural nets performing algorithmic and logical tasks

It is argued that the genetic information necessary to encode an algorithmic neural processor tutoring an otherwise randomly connected biological neural net is represented by the entropy of the analogous minimal Turing machine. Such a near-minimal machine is constructed performing the whole range of bivalent propositional logic in variables. Neural nets computing the same task are presented; their informational entropy can be gauged with reference to the analogous Turing machine. It is also shown that nets with one hidden layer can be trained to perform algorithms solving propositional logic by error back-propagation. Received: 30 June 1995 / Accepted in revised form: 9 January 1996     

Efficiency of neural networks for prediction of in vitro culture conditions and inoculum properties for optimum productivity

This study represents an ANN based computational scheming of physical, chemical and biological parameters at flask level for mass multiplication of plants through micropropagation using bioreactors of larger volumes. The optimal culture environment at small scale for Glycyrrhiza plant was predicted by using neural network approach in terms of pH and volume of growth medium per culture flask, incubation room temperature and month of inoculation along with inoculum properties in terms of inoculum size, fresh weight and number of explant per flask. This kind of study could be a model system in commercial propagation of various economically important plants in bioreactors using tissue culture technique. In present course of study the ANN was trained by implementing MATLAB neural network. A feed-forward back propagation type network was created for input vector (seven input elements), with single hidden layer (seven nodes) and one output unit in output layer. The ‘tansig’ and ‘purelin’ transfer functions were adapted for hidden and output layers respectively. The four training functions viz. traingda, trainrp, traincgf, traincgb were randomly selected to train four networks which further examined with available dataset. The efficiency of neural networks was concluded by the comparison of results obtained from this study with that of empirical data obtained from the detailed tissue culture experiments and designated as Target set (mean fresh weight biomass per culture flask after 40 days of in vitro culture duration). Efficiency of networks for better training initialization was judged on the basis of comparative analysis of ‘Mean Square Error at zero epoch’ for each network trained in which the least error at initial point was observed with trainrp followed by traincgb and traincgf. A comparative assessment between experimental target data range obtained from wet lab practice and all trained network output range for the efficiency of trained networks for least deviation from target range revealed the output range of network ‘trainrp’ was closest to the empirical target range while least comparison was worked out from network ‘traincgb’ which had output range more than the target decided and ultimately showed meaningless result.     

Intelligent scheduling of FMSs with inductive learning capability using neural networks

With the growing uncertainty and complexity in the manufacturing environment, most scheduling problems have been proven to be NP-complete and this can degrade the performance of conventional operations research (OR) techniques. This article presents a system-attribute-oriented knowledge-based scheduling system (SAOSS) with inductive learning capability. With the rich heritage from artificial intelligence (AI), SAOSS takes a multialgorithm paradigm which makes it more intelligent, flexible, and suitable than others for tackling complicated, dynamic scheduling problems. SAOSS employs an efficient and effective inductive learning method, a continuous iterative dichotomister 3 (CID3) algorithm, to induce decision rules for scheduling by converting corresponding decision trees into hidden layers of a self-generated neural network. Connection weights between hidden units imply the scheduling heuristics, which are then formulated into scheduling rules. An FMS scheduling problem is also given for illustration. The scheduling results show that the system-attribute-oriented knowledge-based approach is capable of addressing dynamic scheduling problems.     

Best harmony, unified RPCL and automated model selection for unsupervised and supervised learning on Gaussian mixtures, three-layer nets and ME-RBF-SVM models

After introducing the fundamentals of BYY system and harmony learning, which has been developed in past several years as a unified statistical framework for parameter learning, regularization and model selection, we systematically discuss this BYY harmony learning on systems with discrete inner-representations. First, we shown that one special case leads to unsupervised learning on Gaussian mixture. We show how harmony learning not only leads us to the EM algorithm for maximum likelihood (ML) learning and the corresponding extended KMEAN algorithms for Mahalanobis clustering with criteria for selecting the number of Gaussians or clusters, but also provides us two new regularization techniques and a unified scheme that includes the previous rival penalized competitive learning (RPCL) as well as its various variants and extensions that performs model selection automatically during parameter learning. Moreover, as a by-product, we also get a new approach for determining a set of 'supporting vectors' for Parzen window density estimation. Second, we shown that other special cases lead to three typical supervised learning models with several new results. On three layer net, we get (i) a new regularized ML learning, (ii) a new criterion for selecting the number of hidden units, and (iii) a family of EM-like algorithms that combines harmony learning with new techniques of regularization. On the original and alternative models of mixture-of-expert (ME) as well as radial basis function (RBF) nets, we get not only a new type of criteria for selecting the number of experts or basis functions but also a new type of the EM-like algorithms that combines regularization techniques and RPCL learning for parameter learning with either least complexity nature on the original ME model or automated model selection on the alternative ME model and RBF nets. Moreover, all the results for the alternative ME model are also applied to other two popular nonparametric statistical approaches, namely kernel regression and supporting vector machine. Particularly, not only we get an easily implemented approach for determining the smoothing parameter in kernel regression, but also we get an alternative approach for deciding the set of supporting vectors in supporting vector machine.     

肿瘤亚型识别研究中智能算法的应用

目的:为解决肿瘤亚型识别过程中易出现的维数灾难和过拟合问题,提出了一种改进的粒子群BP神经网络集成算法。方法:算法采用欧式距离和互信息来初步过滤冗余基因,之后用Relief算法进一步处理,得到候选特征基因集合。采用BP神经网络作为基分类器,将特征基因提取与分类器训练相结合,改进的粒子群对其权值和阈值进行全局搜索优化。结果:当隐含层神经元个数为5时,候选特征基因个数为110时,QPSO/BP算法全局优化和搜索,此时的分类准确率最高。结论:该算法不但提高了肿瘤分型识别的准确率,而且降低了学习的复杂度。     

A Lempel-Ziv complexity-based neural network pruning algorithm

This paper presents a pruning method for artificial neural networks (ANNs) based on the 'Lempel-Ziv complexity' (LZC) measure. We call this method the 'silent pruning algorithm' (SPA). The term 'silent' is used in the sense that SPA prunes ANNs without causing much disturbance during the network training. SPA prunes hidden units during the training process according to their ranks computed from LZC. LZC extracts the number of unique patterns in a time sequence obtained from the output of a hidden unit and a smaller value of LZC indicates higher redundancy of a hidden unit. SPA has a great resemblance to biological brains since it encourages higher complexity during the training process. SPA is similar to, yet different from, existing pruning algorithms. The algorithm has been tested on a number of challenging benchmark problems in machine learning, including cancer, diabetes, heart, card, iris, glass, thyroid, and hepatitis problems. We compared SPA with other pruning algorithms and we found that SPA is better than the 'random deletion algorithm' (RDA) which prunes hidden units randomly. Our experimental results show that SPA can simplify ANNs with good generalization ability.     

Pyrolysis patterns of 5 close Corynebacterium species analyzed by artificial neural networks

In the present study, an artificial neural network was trained with the Stuttgart Neural Networks Simulator, in order to identify Corynebacterium species by analyzing their pyrolysis patterns. An earlier study described the combination of pyrolysis, gas chromatography and atomic emission detection we used on whole cell bacteria. Carbon, sulfur and nitrogen were detected in the pyrolysis compounds. Pyrolysis patterns were obtained from 52 Corynebacterium strains belonging to 5 close species. These data were previously analyzed by Euclidean distances calculation followed by Unweighted Pair Group Method of Averages, a clustering method. With this early method, strains from 3 of the 5 species (C. xerosis, C. freneyi and C. amycolatum) were correctly characterized even if the 29 strains of C. amycolatum were grouped into 2 subgroups. Strains from the 2 remaining species (C. minutissimum and C. striatum) cannot be separated. To build an artificial neural network, able to discriminate the 5 previous species, the pyrolysis data of 42 selected strains were used as learning set and the 10 remaining strains as testing set. The chosen learning algorithm was Back-Propagation with Momentum. Parameters used to train a correct network are described here, and the results analyzed. The obtained artificial neural network has the following cone-shaped structure: 144 nodes in input, 25 and 9 nodes in 2 successive hidden layers, and then 5 outputs. It could classify all the strains in their species group. This network completes a chemotaxonomic method for Corynebacterium identification.     

基于神经网络简单集成的湖库富营养化综合评价模型

根据中国水利部推荐的地表水富营养化控制标准,以叶绿素a、总磷、总氮、化学需氧量和透明度为评价指标,采用线性插值方法生成均匀分布的训练样本,建立了用于湖泊、水库富营养化综合评价的神经网络简单集成模型,其个体网络采用反向传播网络。通过递增法分别确定个体网络隐含层节点数为3,集成规模为40。所有个体网络均采用弹性反传训练算法和带动量的梯度下降学习算法。将该模型应用于巢湖富营养化综合评价,结果表明该模型有效消除了单个反向传播神经网络对初始网络权重的敏感性,泛化能力得到显著的提高。该模型的评价结果与综合营养状态指数法差异极显著,而与插值评分法差异不显著;但相关性较高,相关系数分别为0．9406和0．8891。通过对比分析,表明该模型较好地归纳了评价标准中的潜在评价规则,评价结果客观、可靠。     

A Novel Approach for Lie Detection Based on F-Score and Extreme Learning Machine

A new machine learning method referred to as F-score_ELM was proposed to classify the lying and truth-telling using the electroencephalogram (EEG) signals from 28 guilty and innocent subjects. Thirty-one features were extracted from the probe responses from these subjects. Then, a recently-developed classifier called extreme learning machine (ELM) was combined with F-score, a simple but effective feature selection method, to jointly optimize the number of the hidden nodes of ELM and the feature subset by a grid-searching training procedure. The method was compared to two classification models combining principal component analysis with back-propagation network and support vector machine classifiers. We thoroughly assessed the performance of these classification models including the training and testing time, sensitivity and specificity from the training and testing sets, as well as network size. The experimental results showed that the number of the hidden nodes can be effectively optimized by the proposed method. Also, F-score_ELM obtained the best classification accuracy and required the shortest training and testing time.     

Modelling and simulation of signal transductions in an apoptosis pathway by using timed Petri nets

This paper first presents basic Petri net components representing molecular interactions and mechanisms of signalling pathways, and introduces a method to construct a Petri net model of a signalling pathway with these components. Then a simulation method of determining the delay time of transitions, by using timed Petri nets — i.e. the time taken in firing of each transition — is proposed based on some simple principles that the number of tokens flowed into a place is equivalent to the number of tokens flowed out. Finally, the availability of proposed method is confirmed by observing signalling transductions in biological pathways through simulation experiments of the apoptosis signalling pathways as an example.     

Use of neural networks in the mathematical modelling of the enzymic synthesis of amoxicillin catalysed by penicillin G acylase immobilized in chitosan

This study focuses in the mathematical modelling of the enzymic synthesis of amoxicillin by the reaction of p-hydroxyphenylglycine methyl ester and 6-aminopenicillanic acid (6APA), catalyzed by penicillin G acylase (PGA) immobilized on glutaraldehyde-chitosan, at 25°C and pH 6.5. Previous work on the kinetics and mechanism of reaction showed that the use of neural networks seems to be an interesting alternative to simulate experimental data of antibiotic production. Therefore, two feedforward neural networks, with one hidden layer, were trained and used to forecast the rates of amoxicillin and p-hydroxyphenylglycine (POHPG) net production. First of all, some parameters that affect the network performed were investigated, such as the number of nodes between the input and hidden layers and the number of interactions during the learning phase. Afterwards, hybrid models that coupled artificial neural networks to mass-balance equations were used to reproduce the performance of batch reactors for the production of amoxicillin. This approach provided accurate results, within the range of substrate concentration studied.     

An improvement of extreme learning machine for compact single-hidden-layer feedforward neural networks

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing.     

Contribution to the theory of random and biased nets

The probabilistic theory of random and biased nets is further developed by the “tracing” method treated previously. A number of biases expected to be operating in nets, particularly in sociograms, is described. Distribution of closed chain lengths is derived for random nets and for nets with a simple “reflexive” bias. The “island model” bias is treated for the case of two islands and a single axon tracing, resulting in a pair of linear difference equations with two indices. The reflexive bias is extended to multiple-axon tracing by an approximate method resulting in a modification of the random net recursion formula. Results previously obtained are compared with empirical findings and attempts are made to account for observed discrepancies.     

Empirical evaluation of a dynamic experiment design method for prediction of MHC class I-binding peptides

The ability to predict MHC-binding peptides remains limited despite ever expanding demands for specific immunotherapy against cancers, infectious diseases, and autoimmune disorders. Previous analyses revealed position-specific preference of amino acids but failed to detect sequence patterns. Efforts to use computational analysis to identify sequence patterns have been hampered by the insufficiency of the number/quality of the peptide binding data. We propose here a dynamic experiment design to search for sequence patterns that are common to the MHC class I-binding peptides. The method is based on a committee-based framework of query learning using hidden Markov models as its component algorithm. It enables a comprehensive search of a large variety (20(9)) of peptides with a small number of experiments. The learning was conducted in seven rounds of feedback loops, in which our computational method was used to determine the next set of peptides to be analyzed based on the results of the earlier iterations. After these training cycles, the algorithm enabled a real number prediction of MHC binding peptides with an accuracy surpassing that of the hitherto best performing positional scanning method.