A cascading structure and training method for multilayer neural networks

A new structure and training method for multilayer neural networks is presented. The proposed method is based on cascade training of subnetworks and optimizing weights layer by layer. The training procedure is completed in two steps. First, a subnetwork, m inputs and n outputs as the style of training samples, is trained using the training samples. Secondly the outputs of the subnetwork is taken as the inputs and the outputs of the training sample as the desired outputs, another subnetwork with n inputs and n outputs is trained. Finally the two trained subnetworks are connected and a trained multilayer neural networks is created. The numerical simulation results based on both linear least squares back-propagation (LSB) and traditional back-propagation (BP) algorithm have demonstrated the efficiency of the proposed method.     

An integrated approach to optimization of Escherichia coli fermentations using historical data

Using a fermentation database for Escherichia coli producing green fluorescent protein (GFP), we have implemented a novel three-step optimization method to identify the process input variables most important in modeling the fermentation, as well as the values of those critical input variables that result in an increase in the desired output. In the first step of this algorithm, we use either decision-tree analysis (DTA) or information theoretic subset selection (ITSS) as a database mining technique to identify which process input variables best classify each of the process outputs (maximum cell concentration, maximum product concentration, and productivity) monitored in the experimental fermentations. The second step of the optimization method is to train an artificial neural network (ANN) model of the process input-output data, using the critical inputs identified in the first step. Finally, a hybrid genetic algorithm (hybrid GA), which includes both gradient and stochastic search methods, is used to identify the maximum output modeled by the ANN and the values of the input conditions that result in that maximum. The results of the database mining techniques are compared, both in terms of the inputs selected and the subsequent ANN performance. For the E. coli process used in this study, we identified 6 inputs from the original 13 that resulted in an ANN that best modeled the GFP fluorescence outputs of an independent test set. Values of the six inputs that resulted in a modeled maximum fluorescence were identified by applying a hybrid GA to the ANN model developed. When these conditions were tested in laboratory fermentors, an actual maximum fluorescence of 2.16E6 AU was obtained. The previous high value of fluorescence that was observed was 1.51E6 AU. Thus, this input condition set that was suggested by implementing the proposed optimization scheme on the available historical database increased the maximum fluorescence by 55%.     

Retrospective optimization of time-dependent fermentation control strategies using time-independent historical data

We have previously shown the usefulness of historical data for fermentation process optimization. The methodology developed includes identification of important process inputs, training of an artificial neural network (ANN) process model, and ultimately use of the ANN model with a genetic algorithm to find the optimal values of each critical process input. However, this approach ignores the time-dependent nature of the system, and therefore, does not fully utilize the available information within a database. In this work, we propose a method for incorporating time-dependent optimization into our previously developed three-step optimization routine. This is achieved by an additional step that uses a fermentation model (consisting of coupled ordinary differential equations (ODE)) to interpret important time-course features of the collected data through adjustments in model parameters. Important process variables not explicitly included in the model were then identified for each model parameter using automatic relevance determination (ARD) with Gaussian process (GP) models. The developed GP models were then combined with the fermentation model to form a hybrid neural network model that predicted the time-course activity of the cell and protein concentrations of novel fermentation conditions. A hybrid-genetic algorithm was then used in conjunction with the hybrid model to suggest optimal time-dependent control strategies. The presented method was implemented upon an E. coli fermentation database generated in our laboratory. Optimization of two different criteria (final protein yield and a simplified economic criteria) was attempted. While the overall protein yield was not increased using this methodology, we were successful in increasing a simplified economic criterion by 15% compared to what had been previously observed. These process conditions included using 35% less arabinose (the inducer) and 33% less typtone in the media and reducing the time required to reach the maximum protein concentration by 10% while producing approximately the same level of protein as the previous optimum.     

Neural network modeling in optimisation of continuous fermentation processes

The capability of self-recurrent neural networks in dynamic modeling of continuous fermentation is investigated in this simulation study. In the past, feedforward neural networks have been successfully used as one-step-ahead predictors. However, in steady-state optimisation of continuous fermentations the neural network model has to be iterated to predict many time steps ahead into the future in order to get steady-state values of the variables involved in objective cost function, and this iteration may result in increasing errors. Therefore, as an alternative to classical feedforward neural network trained by using backpropagation method, self-recurrent multilayer neural net trained by backpropagation through time method was chosen in order to improve accuracy of long-term predictions. Prediction capabilities of the resulting neural network model is tested by implementing this into the Integrated System Optimisation and Parameter Estimation (ISOPE) optimisation algorithm. Maximisation of cellular productivity of the baker's yeast continuous fermentation was used as the goal of the proposed optimising control problem. The training and prediction results of proposed neural network and performances of resulting optimisation structure are demonstrated.     

Vision-based force measurement using neural networks for biological cell microinjection

This paper presents a vision-based force measurement method using an artificial neural network model. The proposed model is used for measuring the applied load to a spherical biological cell during micromanipulation process. The devised vision-based method is most useful when force measurement capability is required, but it is very challenging or even infeasible to use a force sensor. Artificial neural networks in conjunction with image processing techniques have been used to estimate the applied load to a cell. A bio-micromanipulation system capable of force measurement has also been established in order to collect the training data required for the proposed neural network model. The geometric characterization of zebrafish embryos membranes has been performed during the penetration of the micropipette prior to piercing. The geometric features are extracted from images using image processing techniques. These features have been used to describe the shape and quantify the deformation of the cell at different indentation depths. The neural network is trained by taking the visual data as the input and the measured corresponding force as the output. Once the neural network is trained with sufficient number of data, it can be used as a precise sensor in bio-micromanipulation setups. However, the proposed neural network model is applicable for indentation of any other spherical elastic object. The results demonstrate the capability of the proposed method. The outcomes of this study could be useful for measuring force in biological cell micromanipulation processes such as injection of the mouse oocyte/embryo.     

Optimization of fed-batch bioreactor using neural network model

An algorithm using feedforward neural network model for determining optimal substrate feeding policies for fed-batch fermentation process is presented in this work. The algorithm involves developing the neural network model of the process using the sampled data. The trained neural network model in turn is used for optimization purposes. The advantages of this technique is that optimization can be achieved without detailed kinetic model of the process and the computation of gradient of objective function with respect to control variables is straightforward. The application of the technique is demonstrated with two examples, namely, production of secreted protein and invertase. The simulation results show that the discrete-time dynamics of fed-batch bioreactor can be satisfactorily approximated using a feedforward sigmoidal neural network. The optimal policies obtained with the neural network model agree reasonably well with the previously reported results.     

Optimization of a fermentation medium using neural networks and genetic algorithms

Artificial neural networks and genetic algorithms are used to model and optimize a fermentation medium for the production of the enzyme hydantoinase by Agrobacterium radiobacter. Experimental data reported in the literature were used to build two neural network models. The concentrations of four medium components served as inputs to the neural network models, and hydantoinase or cell concentration served as a single output of each model. Genetic algorithms were used to optimize the input space of the neural network models to find the optimum settings for maximum enzyme and cell production. Using this procedure, two artificial intelligence techniques have been effectively integrated to create a powerful tool for process modeling and optimization.     

条件联想神经网络研究

在人脑的某些功能和神经系统中的突前抑制机制启发下,本文提出一个新型的神经网络模型——条件联想神经网络.模型是一个有突触前抑制的联想记忆神经网络.通过初步分析和计算机模拟,证明本模型具有一般联想记忆模型所未有的一些新的特性,如可以在不同条件下,对同一输入有不同的反应.对同一输入,在不同的条件下,又可以有相同的反应.这些特点将有助于人们对神经系统中信息处理过程的了解.此外,文中也指出可能实现本模型的神经结构.     

Training back-propagation neural networks to define and detect DNA-binding sites

A three layered back-propagation neural network was trained to recognize E. coli promoters of the 17 base spacing class. To this end, the network was presented with 39 promoter sequences and derivatives of those sequences as positive inputs; 60% A + T random sequences and sequences containing 2 promoter-down point mutations were used as negative inputs. The entire promoter sequence of 58 bases, approximately -50 to +8, was entered as input. The network was asked to associate an output of 1.0 with promoter sequence input and 0.0 with non-promoter input. Generally, after 100,000 input cycles, the network was virtually perfect in classifying the training set. A trained network was about 80% effective in recognizing 'new' promoters which were not in the training set, with a false positive rate below 0.1%. Network searches on pBR322 and on the lambda genome were also performed. Overall the results were somewhat better than the best rule-based procedures. The trained network can be analyzed both for its choice of base and relative weighting, positive and negative, at each position of the sequence. This method, which requires only appropriate input/output training pairs, can be used to define and search for any DNA regulatory sequence for which there are sufficient exemplars.     

Neural network programming in bioprocess variable estimation and state prediction.

A neural network program with efficient learning ability for bioprocess variable estimation and state prediction was developed. A 3 layer, feed-forward neural network architecture was used, and the program was written in Quick C ver 2.5 for an IBM compatible computer with a 80486/33 MHz processor. A back propagation training algorithm was used based on learning by pattern and momentum in a combination as used to adjust the connection of weights of the neurons in adjacent layers. The delta rule was applied in a gradient descent search technique to minimize a cost function equal to the mean square difference between the target and the network output. A non-linear, sigmoidal logistic transfer function was used in squashing the weighted sum of the inputs of each neuron to a limited range output. A good neural network prediction model was obtained by training with a sequence of past time course data of a typical bioprocess. The well trained neural network estimated accurately and rapidly the state variables with or without noise even under varying process dynamics.     

Artificial intelligence based optimization of exocellular glucansucrase production from Leuconostoc dextranicum NRRL B-1146

Two different artificial intelligence techniques namely artificial neural network (ANN) and genetic algorithm (GA) were integrated for optimizing fermentation medium for the production of glucansucrase. The experimental data reported in a previous study were used to build the neural network. The ANN was trained using the back propagation algorithm. The ANN predicted values showed good agreement with the experimentally reported ones from a response surface based experiment. The concentrations of three medium components: viz Tween 80, sucrose and K(2)HPO(4) served as inputs to the neural network model and the enzyme activity as the output of the model. A model was generated with a coefficient of correlation (R(2)) of 1.0 for the training set and 0.90 for the test data. A genetic algorithm was used to optimize the input space of the neural network model to find the optimum settings for maximum enzyme activity. This artificial neural network supported genetic algorithm predicted a maximum glucansucrase activity of 6.92U/ml at medium composition of 0.54% (v/v) Tween 80, 5.98% (w/v) sucrose and 1.01% (w/v) K(2)HPO(4). ANN-GA predicted model gave a 6.0% increase of enzyme activity over the regression based prediction for optimized enzyme activity. The maximum enzyme activity experimentally obtained using the ANN-GA designed medium was 6.75+/-0.09U/ml which was in good agreement with the predicted value.     

Dynamic neural networks as a tool for the online optimization of industrial fermentation

A system for online optimization of industrial fermentation based on a model with dynamic neural networks is described. The developed dynamic neural network, consisting of adapted neurons to consider the process dynamics, can model the complex, non-linear fermentation of beer in order to predict the future process. The predicted trajectories of gravity, pH, and diacetyl are in agreement with the experimental data measured at an automated pilot fermenter. It was possible to predict the future course of the batch fermentation as soon as 12 h of process data were available. In combination with the variational principle, the process model was used to optimize productivity. The temperature trajectory is optimized using a cost functional, including technical and technological conditions of the brewery in order to reduce the process time by steady product quality. The results show a reduction of the process time of up to 20%, which leads to an increase in utilization capacity.     

Application of neural computing methods for interpreting phospholipid fatty acid profiles of natural microbial communities

The microbial community compositions of surface and subsurface marine sediments and sediments lining burrows of marine polychaetes and hemichordates from the North Inlet estuary (near Georgetown, S.C. ) were analyzed by comparing ester-linked phospholipid fatty acid (PLFA) profiles with a back-propagating neural network (NN). The NNs were trained to relate PLFA inputs to sediment type outputs (e.g., surface, subsurface, and burrow lining) and worm species (e.g., Notomastus lobatus, Balanoglossus aurantiacus, and Branchyoasychus americana). Sensitivity analysis was used to determine which of the 60 PLFAs significantly contributed to training the NN. The NN architecture was optimized by changing the number of hidden neurons and calculating the cross-validation error between predicted and actual outputs of training and test data. The optimal NN architecture was found to be four hidden neurons with 60-input neurons representing the 60 PLFAs, and four output neurons coding for both sediment types and worm species. Comparison of cross-validation results using NNs and linear discriminant analysis (LDA) revealed that NNs had significantly fewer incorrect classifications (2.7%) than LDA (8.4%). For the NN cross-validation, both sediment type and worm species had 3 incorrect classifications out of 112. For the LDA cross-validation, sediment type and worm species had 7 and 12 incorrect classifications out of 112, respectively. Sensitivity analysis of the trained NNs revealed that 17 fatty acids explained 50% of variability in the data set. These PLFAs were highly different among sediments and burrow types, indicating significant differences in the microbiota.     

Using neural networks as an aid in the determination of disease status: comparison of clinical diagnosis to neural-network predictions in a pedigree with autosomal dominant limb-girdle muscular dystrophy.

Studies of the genetics of certain inherited diseases require expertise in the determination of disease status even for single-locus traits. For example, in the diagnosis of autosomal dominant limb-girdle muscular dystrophy (LGMD1A), it is not always possible to make a clear-cut determination of disease, because of variability in the diagnostic criteria, age at onset, and differential presentation of disease. Mapping such diseases is greatly simplified if the data present a homogeneous genetic trait and if disease status can be reliably determined. Here, we present an approach to determination of disease status, using methods of artificial neural-network analysis. The method entails "training" an artificial neural network, with input facts (based on diagnostic criteria) and related results (based on disease diagnosis). The network contains weight factors connecting input "neurons" to output "neurons," and these connections are adjusted until the network can reliably produce the appropriate outputs for the given input facts. The trained network can be "tested" with a second set of facts, in which the outcomes are known but not provided to the network, to see how well the training has worked. The method was applied to members of a pedigree with LGMD1A, now mapped to chromosome 5q. We used diagnostic criteria and disease status to train a neural network to classify individuals as "affected" or "not affected." The trained network reproduced the disease diagnosis of all individuals of known phenotype, with 98% reliability. This approach defined an appropriate choice of clinical factors for determination of disease status. Additionally, it provided insight into disease classification of those considered to have an "unknown" phenotype on the basis of standard clinical diagnostic methods.     

On a recurrent neural network producing oscillations

A recurrent two-node neural network producing oscillations is analyzed. The network has no true inputs and the outputs from the network exhibit a circular phase portrait. The weight configuration of the network is investigated, resulting in analytical weight expressions, which are compared with numerical weight estimates obtained by training the network on the desired trajectories. The values predicted by the analytical expressions agree well with the findings from the numerical study, and can also explain the asymptotic properties of the networks studied.     

Reconstructing muscle activation during normal walking: a comparison of symbolic and connectionist machine learning techniques

One symbolic (rule-based inductive learning) and one connectionist (neural network) machine learning technique were used to reconstruct muscle activation patterns from kinematic data measured during normal human walking at several speeds. The activation patterns (or desired outputs) consisted of surface electromyographic (EMG) signals from the semitendinosus and vastus medialis muscles. The inputs consisted of flexion and extension angles measured at the hip and knee of the ipsilateral leg, their first and second derivatives, and bilateral foot contact information. The training set consisted of data from six trials, at two different speeds. The testing set consisted of data from two additional trials (one at each speed), which were not in the training set. It was possible to reconstruct the muscular activation at both speeds using both techniques. Timing of the reconstructed signals was accurate. The integrated value of the activation bursts was less accurate. The neural network gave a continuous output, whereas the rule-based inductive learning rule tree gave a quantised activation level. The advantage of rule-based inductive learning was that the rules used were both explicit and comprehensible, whilst the rules used by the neural network were implicit within its structure and not easily comprehended. The neural network was able to reconstruct the activation patterns of both muscles from one network, whereas two separate rule sets were needed for the rule-based technique. It is concluded that machine learning techniques, in comparison to explicit inverse muscular skeletal models, show good promise in modelling nearly cyclic movements such as locomotion at varying walking speeds. However, they do not provide insight into the biomechanics of the system, because they are not based on the biomechanical structure of the system.     

An olfactory neuronal network for vapor recognition in an artificial nose

Odorant sensitivity and discrimination in the olfactory system appear to involve extensive neural processing of the primary sensory inputs from the olfactory epithelium. To test formally the functional consequences of such processing, we implemented in an artificial chemosensing system a new analytical approach that is based directly on neural circuits of the vertebrate olfactory system. An array of fiber-optic chemosensors, constructed with response properties similar to those of olfactory sensory neurons, provide time-varying inputs to a computer simulation of the olfactory bulb (OB). The OB simulation produces spatiotemporal patterns of neuronal firing that vary with vapor type. These patterns are then recognized by a delay line neural network (DLNN). In the final output of these two processing steps, vapor identity is encoded by the spatial patterning of activity across units in the DLNN, and vapor intensity is encoded by response latency. The OB-DLNN combination thus separates identity and intensity information into two distinct codes carried by the same output units, enabling discrimination among organic vapors over a range of input signal intensities. In addition to providing a well-defined system for investigating olfactory information processing, this biologically based neuronal network performs better than standard feed-forward neural networks in discriminating vapors when small amounts of training data are used. Received: 30 June 1997 / Accepted in revised form: 12 January 1998     

Hybrid model building methodology using unsupervised fuzzy clustering and supervised neural networks.

This paper suggests a model building methodology for dealing with new processes. The methodology, called Hybrid Fuzzy Neural Networks (HFNN), combines unsupervised fuzzy clustering and supervised neural networks in order to create simple and flexible models. Fuzzy clustering was used to define relevant domains on the input space. Then, sets of multilayer perceptrons (MLP) were trained (one for each domain) to map input-output relations, creating, in the process, a set of specified sub-models. The estimated output of the model was obtained by fusing the different sub-model outputs weighted by their predicted possibilities. On-line reinforcement learning enabled improvement of the model. The determination of the optimal number of clusters is fundamental to the success of the HFNN approach. The effectiveness of several validity measures was compared to the generalization capability of the model and information criteria. The validity measures were tested with fermentation simulations and real fermentations of a yeast-like fungus, Aureobasidium pullulans. The results outline the criteria limitations. The learning capability of the HFNN was tested with the fermentation data. The results underline the advantages of HFNN over a single neural network.     

人工神经网络在发酵工业中的应用

人工神经网络技术具有很强的非线性映射能力,用于系统的非线性建模,具有无可比拟的优势,广泛应用于发酵过程中培养基的优化和系统建模与控制方面,本主要介绍了人工神经网络的基本原理与使用方法,以及BP神经网络在非线性函数逼近的优点,详细介绍了其在发酵培养基优化,连续搅拌反应器神经网络估计,分批发酵及补料分批发酵过程建模与控制优化中的应用实例。     

An evaluation of a neural network for the start-up phase of a continuous recombinant fermentation subject to disturbances

Summary A radial basis neural network was applied to a process for glyceraldehyde-3-phosphate dehydrogenase produced by an Escherichia coli strain containing the plasmid pBR Eco gap. A neural network trained with a pure culture predicted the performance of a fermentation containing wild type cells and/or product in the inoculum better than in the reverse case; this is explained. In general, the network learnt the trends in the concentrations of plasmid-containing cells and the recombinant product more accurately than those of wild type cells and the substrate. This similarity with deterministic networks and the good predictability with some training vectors suggests that neural networks can be used to simulate the start-up phase of recombinant fermentations corrupted by disturbances.