On the ability of neural networks to perform generalization by induction

The ability of neural networks to perform generalization by induction is the ability to learn an algorithm without the benefit of complete information about it. We consider the properties of networks and algorithms that determine the efficiency of generalization. These properties are described in quantitative terms. The most effective generalization is shown to be achieved by networks with the least admissible capacity. General conclusions are illustrated by computer simulations for a three-layered neural network. We draw a quantitative comparison between the general equations and specific results reported here and elsewhere.     

The importance of larger data sets for protein secondary structure prediction with neural networks.

A neural network algorithm is applied to secondary structure and structural class prediction for a database of 318 nonhomologous protein chains. Significant improvement in accuracy is obtained as compared with performance on smaller databases. A systematic study of the effects of network topology shows that, for the larger database, better results are obtained with more units in the hidden layer. In a 32-fold cross validated test, secondary structure prediction accuracy is 67.0%, relative to 62.6% obtained previously, without any evolutionary information on the sequence. Introduction of sequence profiles increases this value to 72.9%, suggesting that the two types of information are essentially independent. Tertiary structural class is predicted with 80.2% accuracy, relative to 73.9% obtained previously. The use of a larger database is facilitated by the introduction of a scaled conjugate gradient algorithm for optimizing the neural network. This algorithm is about 10-20 times as fast as the standard steepest descent algorithm.     

动态神经元的网络模型 Ⅰ.模型和算法

从生物神经元接收和处理信息的基本实验事实出发,提出了一种新的神经网络模型。这个模型修改了大多数现有人工神经网络中关于输出函数只反映静态特性的假设,而强调了神经元发放脉冲的动态过程。模型方程分别对应于突触后电位、感受器电位、始段分级电位和轴突上的的脉冲系列,每个方程都具有明确的生理意义。还给出了计算此非线性方程组解的递推算法和程序框图。因此不仅可对本模型作进一步的理论分析,也可在计算机上仿真,并和相应的生物学实验资料进行对照比较。     

动态神经元的网络模型 Ⅰ.模型和算法

从生物神经元接收和处理信息的基本实验事实出发,提出了一种新的神经网络模型。这个模型修改了大多数现有人工神经网络中关于输出函数只反映静态特性的假设,而强调了神经元发放脉冲的动态过程。模型方程分别对应于突触后电位、感受器电位、始段分级电位和轴突上的的脉冲系列,每个方程都具有明确的生理意义。还给出了计算此非线性方程组解的递推算法和程序框图。因此不仅可对本模型作进一步的理论分析,也可在计算机上仿真,并和相应的生物学实验资料进行对照比较。     

Functional nodes in dynamic neural networks for bioprocess modelling

This contribution presents a novel method for the direct integration of a-priori knowledge in a neural network and its application for the online determination of a secondary metabolite during industrial yeast fermentation. Hereby, existing system knowledge is integrated in an artificial neural network (ANN) by means of 'functional nodes'. A generalized backpropagation algorithm is presented. For illustration, a set of ordinary differential equations describing the diacetyl formation and degradation during the cultivation is incorporated in a functional node and integrated in a dynamic feedforward neural network in a hybrid manner. The results show that a hybrid modelling approach exploiting available a-priori knowledge and experimental data can considerably outperform a pure data-based modelling approach with respect to robustness, generalization and necessary amount of training data. The number of training sets were decreased by 50%, obtaining the same accuracy as in a conventional approach. All incorrect decisions, according to defined cost criteria obtained with the conventional ANN, were avoided.     

Neural network programming in bioprocess variable estimation and state prediction.

A neural network program with efficient learning ability for bioprocess variable estimation and state prediction was developed. A 3 layer, feed-forward neural network architecture was used, and the program was written in Quick C ver 2.5 for an IBM compatible computer with a 80486/33 MHz processor. A back propagation training algorithm was used based on learning by pattern and momentum in a combination as used to adjust the connection of weights of the neurons in adjacent layers. The delta rule was applied in a gradient descent search technique to minimize a cost function equal to the mean square difference between the target and the network output. A non-linear, sigmoidal logistic transfer function was used in squashing the weighted sum of the inputs of each neuron to a limited range output. A good neural network prediction model was obtained by training with a sequence of past time course data of a typical bioprocess. The well trained neural network estimated accurately and rapidly the state variables with or without noise even under varying process dynamics.     

Improved Neural Networks with Random Weights for Short-Term Load Forecasting

An effective forecasting model for short-term load plays a significant role in promoting the management efficiency of an electric power system. This paper proposes a new forecasting model based on the improved neural networks with random weights (INNRW). The key is to introduce a weighting technique to the inputs of the model and use a novel neural network to forecast the daily maximum load. Eight factors are selected as the inputs. A mutual information weighting algorithm is then used to allocate different weights to the inputs. The neural networks with random weights and kernels (KNNRW) is applied to approximate the nonlinear function between the selected inputs and the daily maximum load due to the fast learning speed and good generalization performance. In the application of the daily load in Dalian, the result of the proposed INNRW is compared with several previously developed forecasting models. The simulation experiment shows that the proposed model performs the best overall in short-term load forecasting.     

A space-time delay neural network for motion recognition and its application to lipreading

Motion recognition has received increasing attention in recent years owing to heightened demand for computer vision in many domains, including the surveillance system, multimodal human computer interface, and traffic control system. Most conventional approaches classify the motion recognition task into partial feature extraction and time-domain recognition subtasks. However, the information of motion resides in the space-time domain instead of the time domain or space domain independently, implying that fusing the feature extraction and classification in the space and time domains into a single framework is preferred. Based on this notion, this work presents a novel Space-Time Delay Neural Network (STDNN) capable of handling the space-time dynamic information for motion recognition. The STDNN is unified structure, in which the low-level spatiotemporal feature extraction and high-level space-time-domain recognition are fused. The proposed network possesses the spatiotemporal shift-invariant recognition ability that is inherited from the time delay neural network (TDNN) and space displacement neural network (SDNN), where TDNN and SDNN are good at temporal and spatial shift-invariant recognition, respectively. In contrast to multilayer perceptron (MLP), TDNN, and SDNN, STDNN is constructed by vector-type nodes and matrix-type links such that the spatiotemporal information can be accurately represented in a neural network. Also evaluated herein is the performance of the proposed STDNN via two experiments. The moving Arabic numerals (MAN) experiment simulates the object's free movement in the space-time domain on image sequences. According to these results, STDNN possesses a good generalization ability with respect to the spatiotemporal shift-invariant recognition. In the lipreading experiment, STDNN recognizes the lip motions based on the inputs of real image sequences. This observation confirms that STDNN yields a better performance than the existing TDNN-based system, particularly in terms of the generalization ability. In addition to the lipreading application, the STDNN can be applied to other problems since no domain-dependent knowledge is used in the experiment.     

A Predictive Reinforcement Model of Dopamine Neurons for Learning Approach Behavior

A neural network model of how dopamine and prefrontal cortex activity guides short- and long-term information processing within the cortico-striatal circuits during reward-related learning of approach behavior is proposed. The model predicts two types of reward-related neuronal responses generated during learning: (1) cell activity signaling errors in the prediction of the expected time of reward delivery and (2) neural activations coding for errors in the prediction of the amount and type of reward or stimulus expectancies. The former type of signal is consistent with the responses of dopaminergic neurons, while the latter signal is consistent with reward expectancy responses reported in the prefrontal cortex. It is shown that a neural network architecture that satisfies the design principles of the adaptive resonance theory of Carpenter and Grossberg (1987) can account for the dopamine responses to novelty, generalization, and discrimination of appetitive and aversive stimuli. These hypotheses are scrutinized via simulations of the model in relation to the delivery of free food outside a task, the timed contingent delivery of appetitive and aversive stimuli, and an asymmetric, instructed delay response task.     

一种复合神经元网络结构的并行动力学及快速学习算法

提出两种功能互相不同的神经细胞组成的复合神经元网络(CNN)模型;导出一种特殊结构的CNN的并行动力学;而且证明了它的稳定性。在这些结果基础上,得到快速的假逆矩阵学习算法。计算机仿真试验证实学习算法与动力学稳定性的正确性,并表现出良好的容错性能与存储容量。     

一种复合神经元网络结构的并行动力学及快速学习算法

提出两种功能互相不同的神经细胞组成的复合神经元网络(CNN)模型;导出一种特殊结构的CNN的并行动力学;而且证明了它的稳定性。在这些结果基础上,得到快速的假逆矩阵学习算法。计算机仿真试验证实学习算法与动力学稳定性的正确性,并表现出良好的容错性能与存储容量。     

A template matching model for pattern recognition: Self-organization of templates and template matching by a disinhibitory neural network

A template matching model for pattern recognition is proposed. By following a previouslyproposed algorithm for synaptic modification (Hirai, 1980), the template of a stimulus pattern is selforganized as a spatial distribution pattern of matured synapses on the cells receiving modifiable synapses. Template matching is performed by the disinhibitory neural network cascaded beyond the neural layer composed of the cells receiving the modifiable synapses. The performance of the model has been simulated on a digital computer. After repetitive presentations of a stimulus pattern, a cell receiving the modifiable synapses comes to have the template of that pattern. And the cell in the latter layer of the disinhibitory bitory neural network that receives the disinhibitory input from that cell becomes electively sensitive to that pattern. Learning patterns are not restricted by previously learned ones. They can be subset or superset patterns of the ones previously learned. If an unknown pattern is presented to the model, no cell beyond the disinhibitory neural network will respond. However, if previously learned patterns are embedded in that pattern, the cells which have the templates of those patterns respond and are assumed to transmit the information to higher center. The computer simulation also shows that the model can organize a clean template under a noisy environment.     

Entropic analysis and incremental synthesis of multilayered feedforward neural networks

Neural network architecture optimization is often a critical issue, particularly when VLSI implementation is considered. This paper proposes a new minimization method for multilayered feedforward ANNs and an original approach to their synthesis, both based on the analysis of the information quantity (entropy) flowing through the network. A layer is described as an information filter which selects the relevant characteristics until the complete classification is performed. The basic incremental synthesis method, including the supervised training procedure, is derived to design application-tailored neural paradigms with good generalization capability.     

Gibbs Ensemble Calculations with an Equation of State: An Application to Vapor-Liquid Equilibria

Abstract

A new modification of the Gibbs ensemble Monte Carlo computer simulation method for fluid phase equilibria is described. The modification is based on a thermodynamic model for the vapor phase, and uses an equation of state to account for the weak interactions between the vapor phase molecules. Reductions in the computational time by 30–40% as compared to the original Gibbs ensemble method are obtained. The algorithm is applied to Lennard-Jones - (12,6) fluids and their mixtures and the results are in good agreement with results obtained from simulations using the full Gibbs ensemble method.     

Neural-space generalization of a topological transformation

An investigation is performed to assess the generalization capability found in neural network paradigms to approximate a 2-dimensional coordinate (topological) transformation. A developed strategy uses the example to give a physical meaning to what is meant by generalization. The example shows how to use a neural network paradigm to generalize a two-degree of freedom topological transformation from cartesian end-point coordinates to corresponding joint angle coordinates based only on examples of the mapping. The importance of this example is that it provides a clear understanding of how and what a neural network is actually communications and brings a theoretical idea to a useful understanding. When examples characterize the topology, a collective generalization property begins to emerge and the network learns the topology. If the network is presented with additional examples of the transformation, the network can generate the corresponding joint angles to any unseen position, that is, by generalization. It is also significant that the network's generalization property emerges from the network based on very few training examples! Further, the networks power exists with very few neurons. Results suggest the use of the paradigm's generalization capability to provide solutions to unknown or intractable algorithms for applications.     

Learning-Induced Synchronization and Plasticity of a Developing Neural Network

Learning-induced synchronization of a neural network at various developing stages is studied by computer simulations using a pulse-coupled neural network model in which the neuronal activity is simulated by a one-dimensional map. Two types of Hebbian plasticity rules are investigated and their differences are compared. For both models, our simulations show a logarithmic increase in the synchronous firing frequency of the network with the culturing time of the neural network. This result is consistent with recent experimental observations. To investigate how to control the synchronization behavior of a neural network after learning, we compare the occurrence of synchronization for four networks with different designed patterns under the influence of an external signal. The effect of such a signal on the network activity highly depends on the number of connections between neurons. We discuss the synaptic plasticity and enhancement effects for a random network after learning at various developing stages.     

Astrocyte- neuron interaction as a mechanism responsible for generation of neural synchrony: a study based on modeling and experiments

Neural synchronization is considered as an important mechanism for information processing. In addition, based on recent neurophysiologic findings, it is believed that astrocytes regulate the synaptic transmission of neuronal networks. Therefore, the present study focused on determining the functional contribution of astrocytes in neuronal synchrony using both computer simulations and extracellular field potential recordings. For computer simulations, as a first step, a minimal network model is constructed by connecting two Morris-Lecar neuronal models. In this minimal model, astrocyte-neuron interactions are considered in a functional-based procedure. Next, the minimal network is extended and a biologically plausible neuronal population model is developed which considers functional outcome of astrocyte-neuron interactions too. The employed structure is based on the physiological and anatomical network properties of the hippocampal CA1 area. Utilizing these two different levels of modeling, it is demonstrated that astrocytes are able to change the threshold value of transition from synchronous to asynchronous behavior among neurons. In this way, variations in the interaction between astrocytes and neurons lead to the emergence of synchronous/asynchronous patterns in neural responses. Furthermore, population spikes are recorded from CA1 pyramidal neurons in rat hippocampal slices to validate the modeling results. It demonstrates that astrocytes play a primary role in neuronal firing synchronicity and synaptic coordination. These results may offer a new insight into understanding the mechanism by which astrocytes contribute to stabilizing neural activities.     

Hybrid model building methodology using unsupervised fuzzy clustering and supervised neural networks.

This paper suggests a model building methodology for dealing with new processes. The methodology, called Hybrid Fuzzy Neural Networks (HFNN), combines unsupervised fuzzy clustering and supervised neural networks in order to create simple and flexible models. Fuzzy clustering was used to define relevant domains on the input space. Then, sets of multilayer perceptrons (MLP) were trained (one for each domain) to map input-output relations, creating, in the process, a set of specified sub-models. The estimated output of the model was obtained by fusing the different sub-model outputs weighted by their predicted possibilities. On-line reinforcement learning enabled improvement of the model. The determination of the optimal number of clusters is fundamental to the success of the HFNN approach. The effectiveness of several validity measures was compared to the generalization capability of the model and information criteria. The validity measures were tested with fermentation simulations and real fermentations of a yeast-like fungus, Aureobasidium pullulans. The results outline the criteria limitations. The learning capability of the HFNN was tested with the fermentation data. The results underline the advantages of HFNN over a single neural network.     

Disulfide connectivity prediction using secondary structure information and diresidue frequencies

MOTIVATION: We describe a stand-alone algorithm to predict disulfide bond partners in a protein given only the amino acid sequence, using a novel neural network architecture (the diresidue neural network), and given input of symmetric flanking regions of N-terminus and C-terminus half-cystines augmented with residue secondary structure (helix, coil, sheet) as well as evolutionary information. The approach is motivated by the observation of a bias in the secondary structure preferences of free cysteines and half-cystines, and by promising preliminary results we obtained using diresidue position-specific scoring matrices. RESULTS: As calibrated by receiver operating characteristic curves from 4-fold cross-validation, our conditioning on secondary structure allows our novel diresidue neural network to perform as well as, and in some cases better than, the current state-of-the-art method. A slight drop in performance is seen when secondary structure is predicted rather than being derived from three-dimensional protein structures.     

Association Rate Constants of Ras-Effector Interactions Are Evolutionarily Conserved

Evolutionary conservation of protein interaction properties has been shown to be a valuable indication for functional importance. Here we use homology interface modeling of 10 Ras-effector complexes by selecting ortholog proteins from 12 organisms representing the major eukaryotic branches, except plants. We find that with increasing divergence time the sequence similarity decreases with respect to the human protein, but the affinities and association rate constants are conserved as predicted by the protein design algorithm, FoldX. In parallel we have done computer simulations on a minimal network based on Ras-effector interactions, and our results indicate that in the absence of negative feedback, changes in kinetics that result in similar binding constants have strong consequences on network behavior. This, together with the previous results, suggests an important biological role, not only for equilibrium binding constants but also for kinetics in signaling processes involving Ras-effector interactions. Our findings are important to take into consideration in system biology approaches and simulations of biological networks.