Prediction of Michaelis-Menten constant of beta-glucosidases using nitrophenyl-beta-D-glucopyranoside as substrate

In this study, we attempted to use the neural network to model a quantitative structure-K(m) (Michaelis-Menten constant) relationship for beta-glucosidase, which is an important enzyme to cut the beta-bond linkage in glucose while K(m) is a very important parameter in enzymatic reactions. Eight feedforward backpropagation neural networks with different layers and neurons were applied for the development of predictive model, and twenty-five different features of amino acids were chosen as predictors one by one. The results show that the 20-1 feedforward backpropagation neural network can serve as a predictive model while the normalized polarizability index as well as the amino-acid distribution probability can serve as the predictors. This study threw lights on the possibility of predicting the K(m) in beta-glucosidases based on their amino-acid features.     

Training feedforward neural networks with dynamic particle swarm optimisation

Particle swarm optimisation has been successfully applied to train feedforward neural networks in static environments. Many real-world problems to which neural networks are applied are dynamic in the sense that the underlying data distribution changes over time. In the context of classification problems, this leads to concept drift where decision boundaries may change over time. This article investigates the applicability of dynamic particle swarm optimisation algorithms as neural network training algorithms under the presence of concept drift.     

A probabilistic neural network approach for modeling and classification of bacterial growth/no-growth data

In this paper, we propose to use probabilistic neural networks (PNNs) for classification of bacterial growth/no-growth data and modeling the probability of growth. The PNN approach combines both Bayes theorem of conditional probability and Parzen's method for estimating the probability density functions of the random variables. Unlike other neural network training paradigms, PNNs are characterized by high training speed and their ability to produce confidence levels for their classification decision. As a practical application of the proposed approach, PNNs were investigated for their ability in classification of growth/no-growth state of a pathogenic Escherichia coli R31 in response to temperature and water activity. A comparison with the most frequently used traditional statistical method based on logistic regression and multilayer feedforward artificial neural network (MFANN) trained by error backpropagation was also carried out. The PNN-based models were found to outperform linear and nonlinear logistic regression and MFANN in both the classification accuracy and ease by which PNN-based models are developed.     

Optimization of fed-batch bioreactor using neural network model

An algorithm using feedforward neural network model for determining optimal substrate feeding policies for fed-batch fermentation process is presented in this work. The algorithm involves developing the neural network model of the process using the sampled data. The trained neural network model in turn is used for optimization purposes. The advantages of this technique is that optimization can be achieved without detailed kinetic model of the process and the computation of gradient of objective function with respect to control variables is straightforward. The application of the technique is demonstrated with two examples, namely, production of secreted protein and invertase. The simulation results show that the discrete-time dynamics of fed-batch bioreactor can be satisfactorily approximated using a feedforward sigmoidal neural network. The optimal policies obtained with the neural network model agree reasonably well with the previously reported results.     

Backpropagation neural network predictive control and control scheme comparison of 2,3-butanediol fermentation by Klebsiella oxytoca

A backpropagation neural network (BPN) was applied for the control study of 2,3-butanediol fermentation (2,3-BDL) carried by Klebsiella oxytoca. The measurements of cell mass and glucose were not included in the network models, instead, only the on-line measured product concentrations from the MIMS (membrane introduction mass spectrometer) were involved. Oxygen composition was chosen to be the control variable for this fermentation system for the formation of 2,3-BDL is regulated by oxygen. Oxygen composition was directly correlated to the measured product concentrations. A two-dimensional (number of input nodes by number of data sets) moving window to supply data for on-line, dynamic learning of this fermentation system was applied. The input nodes of the networks were also properly selected. Two neural network control schemes for this 2,3-BDL fermentation were discussed and compared in this work. Fermentations often exist time delay due to the measurement and their slow reaction nature. Hence, the order of time delay for the network controller was also investigated.     

Recurrent neural networks of integrate-and-fire cells simulating short-term memory and wrist movement tasks derived from continuous dynamic networks

Dynamic recurrent neural networks composed of units with continuous activation functions provide a powerful tool for simulating a wide range of behaviors, since the requisite interconnections can be readily derived by gradient descent methods. However, it is not clear whether more realistic integrate-and-fire cells with comparable connection weights would perform the same functions. We therefore investigated methods to convert dynamic recurrent neural networks of continuous units into networks with integrate-and-fire cells. The transforms were tested on two recurrent networks derived by backpropagation. The first simulates a short-term memory task with units that mimic neural activity observed in cortex of monkeys performing instructed delay tasks. The network utilizes recurrent connections to generate sustained activity that codes the remembered value of a transient cue. The second network simulates patterns of neural activity observed in monkeys performing a step-tracking task with flexion/extension wrist movements. This more complicated network provides a working model of the interactions between multiple spinal and supraspinal centers controlling motoneurons. Our conversion algorithm replaced each continuous unit with multiple integrate-and-fire cells that interact through delayed "synaptic potentials". Successful transformation depends on obtaining an appropriate fit between the activation function of the continuous units and the input-output relation of the spiking cells. This fit can be achieved by adapting the parameters of the synaptic potentials to replicate the input-output behavior of a standard sigmoidal activation function (shown for the short-term memory network). Alternatively, a customized activation function can be derived from the input-output relation of the spiking cells for a chosen set of parameters (demonstrated for the wrist flexion/extension network). In both cases the resulting networks of spiking cells exhibited activity that replicated the activity of corresponding continuous units. This confirms that the network solutions obtained through backpropagation apply to spiking networks and provides a useful method for deriving recurrent spiking networks performing a wide range of functions.     

Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway.     

Recruitment learning of boolean functions in sparse random networks

This work presents a new class of neural network models constrained by biological levels of sparsity and weight-precision, and employing only local weight updates. Concept learning is accomplished through the rapid recruitment of existing network knowledge - complex knowledge being realised as a combination of existing basis concepts. Prior network knowledge is here obtained through the random generation of feedforward networks, with the resulting concept library tailored through distributional bias to suit a particular target class. Learning is exclusively local - through supervised Hebbian and Winnow updates - avoiding the necessity for backpropagation of error and allowing remarkably rapid learning. The approach is demonstrated upon concepts of varying difficulty, culminating in the well-known Monks and LED benchmark problems.     

New learning automata based algorithms for adaptation of backpropagation algorithm parameters

One popular learning algorithm for feedforward neural networks is the backpropagation (BP) algorithm which includes parameters, learning rate (eta), momentum factor (alpha) and steepness parameter (lambda). The appropriate selections of these parameters have large effects on the convergence of the algorithm. Many techniques that adaptively adjust these parameters have been developed to increase speed of convergence. In this paper, we shall present several classes of learning automata based solutions to the problem of adaptation of BP algorithm parameters. By interconnection of learning automata to the feedforward neural networks, we use learning automata scheme for adjusting the parameters eta, alpha, and lambda based on the observation of random response of the neural networks. One of the important aspects of the proposed schemes is its ability to escape from local minima with high possibility during the training period. The feasibility of proposed methods is shown through simulations on several problems.     

Bioprocess model building using artificial neural networks

Artificial neural networks are made upon of highly interconnected layers of simple neuron-like nodes. The neurons act as non-linear processing elements within the network. An attractive property of artificial neural networks is that given the appropriate network topology, they are capable of learning and characterising non-linear functional relationships. Furthermore, the structure of the resulting neural network based process model may be considered generic, in the sense that little prior process knowledge is required in its determination. The methodology therefore provides a cost efficient and reliable process modelling technique. One area where such a technique could be useful is biotechnological systems. Here, for example, the use of a process model within an estimation scheme has long been considered an effective means of overcoming inherent on-line measurement problems. However, the development of an accurate process model is extremely time consuming and often results in a model of limited applicability. Artificial neural networks could therefore prove to be a useful model building tool when striving to improve bioprocess operability. Two large scale industrial fermentation systems have been considered as test cases; a fed-batch penicillin fermentation and a continuous mycelial fermentation. Both systems serve to demonstrate the utility, flexibility and potential of the artificial neural network approach to process modelling.     

Implementing Gaussian process inference with neural networks

Gaussian processes compare favourably with backpropagation neural networks as a tool for regression, and Bayesian neural networks have Gaussian process behaviour when the number of hidden neurons tends to infinity. We describe a simple recurrent neural network with connection weights trained by one-shot Hebbian learning. This network amounts to a dynamical system which relaxes to a stable state in which it generates predictions identical to those of Gaussian process regression. In effect an infinite number of hidden units in a feed-forward architecture can be replaced by a merely finite number, together with recurrent connections.     

Backpropagation algorithm adaptation parameters using learning automata

Despite of the many successful applications of backpropagation for training multi-layer neural networks, it has many drawbocks. For complex problems it may require a long time to train the networks, and it may not train at all. Long training time can be the result of the non-optimal parameters. It is not easy to choose appropriate value of the parameters for a particular problem. In this paper, by interconnection of fixed structure learning automata (FSLA) to the feedforward neural networks, we apply learning automata (LA) scheme for adjusting these parameters based on the observation of random response of neural networks. The main motivation in using learning automata as an adaptation algorithm is to use its capability of global optimization when dealing with multi-modal surface. The feasibility of proposed method is shown through simulations on three learning problems: exclusive-or, encoding problem, and digit recognition. The simulation results show that the adaptation of these parameters using this method not only increases the convergence rate of learning but it increases the likelihood of escaping from the local minima.     

Adaptation supports short-term memory in a visual change detection task

The maintenance of short-term memories is critical for survival in a dynamically changing world. Previous studies suggest that this memory can be stored in the form of persistent neural activity or using a synaptic mechanism, such as with short-term plasticity. Here, we compare the predictions of these two mechanisms to neural and behavioral measurements in a visual change detection task. Mice were trained to respond to changes in a repeated sequence of natural images while neural activity was recorded using two-photon calcium imaging. We also trained two types of artificial neural networks on the same change detection task as the mice. Following fixed pre-processing using a pretrained convolutional neural network, either a recurrent neural network (RNN) or a feedforward neural network with short-term synaptic depression (STPNet) was trained to the same level of performance as the mice. While both networks are able to learn the task, the STPNet model contains units whose activity are more similar to the in vivo data and produces errors which are more similar to the mice. When images are omitted, an unexpected perturbation which was absent during training, mice often do not respond to the omission but are more likely to respond to the subsequent image. Unlike the RNN model, STPNet produces a similar pattern of behavior. These results suggest that simple neural adaptation mechanisms may serve as an important bottom-up memory signal in this task, which can be used by downstream areas in the decision-making process.     

Cardiovascular control with artificial neural networks

This paper describes the use of artificial neural networks to model cardiovascular autonomic control in a study of the hemodynamic changes associated with space flight. Cardiovascular system models were created including four parameters: heart rate, contractility, peripheral resistance, and venous tone. Artificial neural networks were then designed and trained. A technique known as backpropagation networking was used and the results of the application of this technique to heart rate control are presented and discussed.     

Control of bioreactors using a neural network model

Control of a continuous bioreactor based on a artificial neural network (ANN) model is carried out theoretically. The ANN model is identified, from input-output data of a bioreactor, using a three-layer feedforward network trained by a back propagation algorithm. The performance of the controller designed on the ANN model is compared with that of a conventional PI controller.     

Modeling of the three-dimensional structure of luffin-α and its simulated reaction with the substrate oligoribonucleotide GAGA

A fundamental problem in biochemistry and molecular biology is understanding the spatial structure of macromolecules and then analyzing their functions. In this study, the three-dimensional structure of a ribosome-inactivating protein luffin-α was predicted using a neural network method and molecular dynamics simulation. A feedforward neural network with the backpropagation learning algorithm were trained on model class of homologous proteins including trichosanthin andα-momorcharin. The distance constraints for the Cα atoms in the protein backbone were utilized to generate a folded crude conformation of luffin-α by model building and the steepest descent minimization approach. The crude conformation was refined by molecular dynamics techniques and a simulated annealing procedure. The interaction between luffin-α and its analogous substrate GAGA was also simulated to understand its action mechanism.     

Using historical data for bioprocess optimization: modeling wine characteristics using artificial neural networks and archived process information

Optimization of fermentation processes is a difficult task that relies on an understanding of the complex effects of processing inputs on productivity and quality outputs. Because of the complexity of these biological systems, traditional optimization methods utilizing mathematical models and statistically designed experiments are less effective, especially on a production scale. At the same time, information is being collected on a regular basis during the course of normal manufacturing and process development that is rarely fully utilized. We are developing an optimization method in which historical process data is used to train an artificial neural network for correlation of processing inputs and outputs. Subsequently, an optimization routine is used in conjunction with the trained neural network to find optimal processing conditions given the desired product characteristics and any constraints on inputs. Wine processing is being used as a case study for this work. Using data from wine produced in our pilot winery over the past 3 years, we have demonstrated that trained neural networks can be used successfully to predict the yeast-fermentation kinetics, as well as chemical and sensory properties of the finished wine, based solely on the properties of the grapes and the intended processing. To accomplish this, a hybrid neural network training method, Stop Training with Validation (STV), has been developed to find the most desirable neural network architecture and training level. As industrial historical data will not be evenly spaced over the entire possible search space, we have also investigated the ability of the trained neural networks to interpolate and extrapolate with data not used during training. Because a company will utilize its own existing process data for this method, the result of this work will be a general fermentation optimization method that can be applied to fermentation processes to improve quality and productivity.     

Training simultaneous recurrent neural network with resilient propagation for static optimization

This paper proposes a non-recurrent training algorithm, resilient propagation, for the Simultaneous Recurrent Neural network operating in relaxation-mode for computing high quality solutions of static optimization problems. Implementation details related to adaptation of the recurrent neural network weights through the non-recurrent training algorithm, resilient backpropagation, are formulated through an algebraic approach. Performance of the proposed neuro-optimizer on a well-known static combinatorial optimization problem, the Traveling Salesman Problem, is evaluated on the basis of computational complexity measures and, subsequently, compared to performance of the Simultaneous Recurrent Neural network trained with the standard backpropagation, and recurrent backpropagation for the same static optimization problem. Simulation results indicate that the Simultaneous Recurrent Neural network trained with the resilient backpropagation algorithm is able to locate superior quality solutions through comparable amount of computational effort for the Traveling Salesman Problem.     

Functional nodes in dynamic neural networks for bioprocess modelling

This contribution presents a novel method for the direct integration of a-priori knowledge in a neural network and its application for the online determination of a secondary metabolite during industrial yeast fermentation. Hereby, existing system knowledge is integrated in an artificial neural network (ANN) by means of 'functional nodes'. A generalized backpropagation algorithm is presented. For illustration, a set of ordinary differential equations describing the diacetyl formation and degradation during the cultivation is incorporated in a functional node and integrated in a dynamic feedforward neural network in a hybrid manner. The results show that a hybrid modelling approach exploiting available a-priori knowledge and experimental data can considerably outperform a pure data-based modelling approach with respect to robustness, generalization and necessary amount of training data. The number of training sets were decreased by 50%, obtaining the same accuracy as in a conventional approach. All incorrect decisions, according to defined cost criteria obtained with the conventional ANN, were avoided.     

Neural network predictive control by MIMS monitored 2,3-butanediol fermentation by Klebsiella oxytoca

With the aid of a membrane introduction mass spectrometer (MIMS), the major product 2,3-butanediol (2,3-BDL) as well as the other metabolites from the fermentation carried by Klebsiella oxytoca can be measured on-line simultaneously. A backpropagation neural network (BPN) being recognized with superior mapping ability was applied to this control study. This neural network adaptive control differs from those conventional controls for fermentation systems in which the measurements of cell mass and glucose are not included in the network model. It is only the measured product concentrations from the MIMS that are involved. Oxygen composition was chosen to be the control variable for this fermentation system. Oxygen composition was directly correlated to the measured product concentrations in the controller model. A two-dimensional (number of input nodes by number of data sets) moving window for on-line, dynamic learning of this fermentation system was applied. The input nodes of the network were also properly selected. Number of the training data sets for obtaining better control results was also determined empirically. Two control structures for this 2,3-BDL fermentation are discussed and compared in this work. The effect from adding time delay element to the network controller was also investigated.