Neural network modeling in optimisation of continuous fermentation processes

The capability of self-recurrent neural networks in dynamic modeling of continuous fermentation is investigated in this simulation study. In the past, feedforward neural networks have been successfully used as one-step-ahead predictors. However, in steady-state optimisation of continuous fermentations the neural network model has to be iterated to predict many time steps ahead into the future in order to get steady-state values of the variables involved in objective cost function, and this iteration may result in increasing errors. Therefore, as an alternative to classical feedforward neural network trained by using backpropagation method, self-recurrent multilayer neural net trained by backpropagation through time method was chosen in order to improve accuracy of long-term predictions. Prediction capabilities of the resulting neural network model is tested by implementing this into the Integrated System Optimisation and Parameter Estimation (ISOPE) optimisation algorithm. Maximisation of cellular productivity of the baker's yeast continuous fermentation was used as the goal of the proposed optimising control problem. The training and prediction results of proposed neural network and performances of resulting optimisation structure are demonstrated.     

Prediction of optimal pH and temperature of cellulases using neural network

Cellulase is an important enzyme widely used in various industries, and now in fermentation of biomass into biofuels. Enzymatic function of cellulase is closely related to pH, temperature, substrate concentration, etc. For newly found cellulase, it would be more cost-effective to predict its optimal pH and temperature before conducting the costly experiments. In this study, we used a 20-2 feedforward backpropagation neural network to build the relationship between information obtained from primary structure of cellulase with optimal pH and temperature to predict the optimal pH and temperature in cellulases. The results show that the amino-acid distribution probability representing the primary structure of cellulase can predict both optimal pH and temperature, whereas various properties of amino acids related to the primary structure cannot do so.     

Predictive models in ecology: Comparison of performances and assessment of applicability

Ecological systems are governed by complex interactions which are mainly nonlinear. In order to capture the inherent complexity and nonlinearity of ecological, and in general biological systems, empirical models recently gained popularity. However, although these models, particularly connectionist approaches such as multilayered backpropagation networks, are commonly applied as predictive models in ecology to a wide variety of ecosystems and questions, there are no studies to date aiming to assess the performance, both in terms of data fitting and generalizability, and applicability of empirical models in ecology. Our aim is hence to provide an overview for nature of the wide range of the data sets and predictive variables, from both aquatic and terrestrial ecosystems with different scales of time-dependent dynamics, and the applicability and robustness of predictive modeling methods on such data sets by comparing different empirical modeling approaches. The models used in this study range from predicting the occurrence of submerged plants in shallow lakes to predicting nest occurrence of bird species from environmental variables and satellite images. The methods considered include k-nearest neighbor (k-NN), linear and quadratic discriminant analysis (LDA and QDA), generalized linear models (GLM) feedforward multilayer backpropagation networks and pseudo-supervised network ARTMAP.Our results show that the predictive performances of the models on training data could be misleading, and one should consider the predictive performance of a given model on an independent test set for assessing its predictive power. Moreover, our results suggest that for ecosystems involving time-dependent dynamics and periodicities whose frequency are possibly less than the time scale of the data considered, GLM and connectionist neural network models appear to be most suitable and robust, provided that a predictive variable reflecting these time-dependent dynamics included in the model either implicitly or explicitly. For spatial data, which does not include any time-dependence comparable to the time scale covered by the data, on the other hand, neighborhood based methods such as k-NN and ARTMAP proved to be more robust than other methods considered in this study. In addition, for predictive modeling purposes, first a suitable, computationally inexpensive method should be applied to the problem at hand a good predictive performance of which would render the computational cost and efforts associated with complex variants unnecessary.     

Rule extraction from minimal neural networks for credit card screening

While feedforward neural networks have been widely accepted as effective tools for solving classification problems, the issue of finding the best network architecture remains unresolved, particularly so in real-world problem settings. We address this issue in the context of credit card screening, where it is important to not only find a neural network with good predictive performance but also one that facilitates a clear explanation of how it produces its predictions. We show that minimal neural networks with as few as one hidden unit provide good predictive accuracy, while having the added advantage of making it easier to generate concise and comprehensible classification rules for the user. To further reduce model size, a novel approach is suggested in which network connections from the input units to this hidden unit are removed by a very straightaway pruning procedure. In terms of predictive accuracy, both the minimized neural networks and the rule sets generated from them are shown to compare favorably with other neural network based classifiers. The rules generated from the minimized neural networks are concise and thus easier to validate in a real-life setting.     

Neuro-fuzzy prediction of uricase production

Recent biotechnology requires implementation of new modelling methods based on knowledge principles and learning structures, comprised in fuzzy knowledge-based systems (FKBS), neural networks (NN) and different hybrid methods. The intelligent modelling approaches solve sufficiently a very important problem - processing of scarce, uncertainty and incomplete numerical and linguistic information about multivariate non-linear and non-stationary systems as well as biotechnological processes. The paper deals with prediction of an enzyme oxidizing uric acid to alantoin - the uricase, produced by Candida utilis 90-12 employing neuro-fuzzy knowledge-based approach. The implemented predictive technique exploits the fact that the fuzzy model can be seen as a network structure, similar to artificial NN, which on computational level assure a high model accuracy. The predictors implemented are four different by nature variables. The developed predictive model shows that best predictors of uricase production are biomass and limiting substrate concentrations.     

Adaptation supports short-term memory in a visual change detection task

The maintenance of short-term memories is critical for survival in a dynamically changing world. Previous studies suggest that this memory can be stored in the form of persistent neural activity or using a synaptic mechanism, such as with short-term plasticity. Here, we compare the predictions of these two mechanisms to neural and behavioral measurements in a visual change detection task. Mice were trained to respond to changes in a repeated sequence of natural images while neural activity was recorded using two-photon calcium imaging. We also trained two types of artificial neural networks on the same change detection task as the mice. Following fixed pre-processing using a pretrained convolutional neural network, either a recurrent neural network (RNN) or a feedforward neural network with short-term synaptic depression (STPNet) was trained to the same level of performance as the mice. While both networks are able to learn the task, the STPNet model contains units whose activity are more similar to the in vivo data and produces errors which are more similar to the mice. When images are omitted, an unexpected perturbation which was absent during training, mice often do not respond to the omission but are more likely to respond to the subsequent image. Unlike the RNN model, STPNet produces a similar pattern of behavior. These results suggest that simple neural adaptation mechanisms may serve as an important bottom-up memory signal in this task, which can be used by downstream areas in the decision-making process.     

The need for stochastic replication of ecological neural networks

Artificial neural networks are becoming increasingly popular as predictive statistical tools in ecosystem ecology and as models of signal processing in behavioural and evolutionary ecology. We demonstrate here that a commonly used network in ecology, the three-layer feed-forward network, trained with the backpropagation algorithm, can be extremely sensitive to the stochastic variation in training data that results from random sampling of the same underlying statistical distribution, with networks converging to several distinct predictive states. Using a random walk procedure to sample error-weight space, and Sammon dimensional reduction of weight arrays, we demonstrate that these different predictive states are not artefactual, due to local minima, but lie at the base of major error troughs in the error-weight surface. We further demonstrate that various gross weight compositions can produce the same predictive state, suggesting the analogy of weight space as a 'patchwork' of multiple predictive states. Our results argue for increased inclusion of stochastic training replication and analysis into ecological and behavioural applications of artificial neural networks.     

Oscillatory burst discharge generated through conditional backpropagation of dendritic spikes.

Gamma frequencies of burst discharge (>40 Hz) have become recognized in select cortical and non-cortical regions as being important in feature extraction, neural synchrony and oscillatory discharge. Pyramidal cells of the electrosensory lateral line lobe (ELL) of Apteronotus leptorhynchus generate burst discharge in relation to specific features of sensory input in vivo that resemble those recognized as gamma frequency discharge when examined in vitro. We have shown that these bursts are generated by an entirely novel mechanism termed conditional backpropagation that involves an intermittent failure of dendritic Na+ spike conduction. Conditional backpropagation arises from a frequency-dependent broadening of dendritic spikes during repetitive discharge, and a mismatch between the refractory periods of somatic and dendritic spikes. A high threshold class of K+ channel, AptKv3.3, is expressed at high levels and distributed over the entire soma-dendritic axis of pyramidal cells. AptKv3.3 channels are shown to contribute to the repolarization of both somatic and dendritic spikes, with pharmacological blockade of dendritic Kv3 channels revealing an important role in controlling the threshold for burst discharge. The entire process of conditional back-propagation and burst output is successfully simulated using a new compartmental model of pyramidal cells that incorporates a cumulative inactivation of dendritic K+ channels during repetitive discharge. This work is important in demonstrating how the success of spike backpropagation can control the output of a principle sensory neuron, and how this process is regulated by the distribution and properties of voltage-dependent K+ channels.     

Functional nodes in dynamic neural networks for bioprocess modelling

This contribution presents a novel method for the direct integration of a-priori knowledge in a neural network and its application for the online determination of a secondary metabolite during industrial yeast fermentation. Hereby, existing system knowledge is integrated in an artificial neural network (ANN) by means of 'functional nodes'. A generalized backpropagation algorithm is presented. For illustration, a set of ordinary differential equations describing the diacetyl formation and degradation during the cultivation is incorporated in a functional node and integrated in a dynamic feedforward neural network in a hybrid manner. The results show that a hybrid modelling approach exploiting available a-priori knowledge and experimental data can considerably outperform a pure data-based modelling approach with respect to robustness, generalization and necessary amount of training data. The number of training sets were decreased by 50%, obtaining the same accuracy as in a conventional approach. All incorrect decisions, according to defined cost criteria obtained with the conventional ANN, were avoided.     

Molecular Simulation of Solution Conformations of the Amyloid β-Peptide Aβ(1–42) by a Backpropagation Neural Network Model

Abstract

The predictive power of solution-dependent conformational states of the Aβ(1–42) peptide of Alzheimer's disease by an optimized backpropagation neural network was tested. It was found that the neural network simulates well the solution-dependent conformations. The model was also examined by using geometry-optimized conformations (hybrid approach of Gasteiger charges plus MM+ molecular-mechanics) where the initial coordinates were obtained by NMR solution spectroscopy.     

Clustering-neural network models for freeway work zone capacity estimation

Two neural network models, called clustering-RBFNN and clustering-BPNN models, are created for estimating the work zone capacity in a freeway work zone as a function of seventeen different factors through judicious integration of the subtractive clustering approach with the radial basis function (RBF) and the backpropagation (BP) neural network models. The clustering-RBFNN model has the attractive characteristics of training stability, accuracy, and quick convergence. The results of validation indicate that the work zone capacity can be estimated by clustering-neural network models in general with an error of less than 10%, even with limited data available to train the models. The clustering-RBFNN model is used to study several main factors affecting work zone capacity. The results of such parametric studies can assist work zone engineers and highway agencies to create effective traffic management plans (TMP) for work zones quantitatively and objectively.     

Modeling of the three-dimensional structure of luffin-α and its simulated reaction with the substrate oligoribonucleotide GAGA

A fundamental problem in biochemistry and molecular biology is understanding the spatial structure of macromolecules and then analyzing their functions. In this study, the three-dimensional structure of a ribosome-inactivating protein luffin-α was predicted using a neural network method and molecular dynamics simulation. A feedforward neural network with the backpropagation learning algorithm were trained on model class of homologous proteins including trichosanthin andα-momorcharin. The distance constraints for the Cα atoms in the protein backbone were utilized to generate a folded crude conformation of luffin-α by model building and the steepest descent minimization approach. The crude conformation was refined by molecular dynamics techniques and a simulated annealing procedure. The interaction between luffin-α and its analogous substrate GAGA was also simulated to understand its action mechanism.     

Network Evolution Induced by Asynchronous Stimuli through Spike-Timing-Dependent Plasticity

In sensory neural system, external asynchronous stimuli play an important role in perceptual learning, associative memory and map development. However, the organization of structure and dynamics of neural networks induced by external asynchronous stimuli are not well understood. Spike-timing-dependent plasticity (STDP) is a typical synaptic plasticity that has been extensively found in the sensory systems and that has received much theoretical attention. This synaptic plasticity is highly sensitive to correlations between pre- and postsynaptic firings. Thus, STDP is expected to play an important role in response to external asynchronous stimuli, which can induce segregative pre- and postsynaptic firings. In this paper, we study the impact of external asynchronous stimuli on the organization of structure and dynamics of neural networks through STDP. We construct a two-dimensional spatial neural network model with local connectivity and sparseness, and use external currents to stimulate alternately on different spatial layers. The adopted external currents imposed alternately on spatial layers can be here regarded as external asynchronous stimuli. Through extensive numerical simulations, we focus on the effects of stimulus number and inter-stimulus timing on synaptic connecting weights and the property of propagation dynamics in the resulting network structure. Interestingly, the resulting feedforward structure induced by stimulus-dependent asynchronous firings and its propagation dynamics reflect both the underlying property of STDP. The results imply a possible important role of STDP in generating feedforward structure and collective propagation activity required for experience-dependent map plasticity in developing in vivo sensory pathways and cortices. The relevance of the results to cue-triggered recall of learned temporal sequences, an important cognitive function, is briefly discussed as well. Furthermore, this finding suggests a potential application for examining STDP by measuring neural population activity in a cultured neural network.     

Criticality enhances the multilevel reliability of stimulus responses in cortical neural networks

Cortical neural networks exhibit high internal variability in spontaneous dynamic activities and they can robustly and reliably respond to external stimuli with multilevel features–from microscopic irregular spiking of neurons to macroscopic oscillatory local field potential. A comprehensive study integrating these multilevel features in spontaneous and stimulus–evoked dynamics with seemingly distinct mechanisms is still lacking. Here, we study the stimulus–response dynamics of biologically plausible excitation–inhibition (E–I) balanced networks. We confirm that networks around critical synchronous transition states can maintain strong internal variability but are sensitive to external stimuli. In this dynamical region, applying a stimulus to the network can reduce the trial-to-trial variability and shift the network oscillatory frequency while preserving the dynamical criticality. These multilevel features widely observed in different experiments cannot simultaneously occur in non-critical dynamical states. Furthermore, the dynamical mechanisms underlying these multilevel features are revealed using a semi-analytical mean-field theory that derives the macroscopic network field equations from the microscopic neuronal networks, enabling the analysis by nonlinear dynamics theory and linear noise approximation. The generic dynamical principle revealed here contributes to a more integrative understanding of neural systems and brain functions and incorporates multimodal and multilevel experimental observations. The E–I balanced neural network in combination with the effective mean-field theory can serve as a mechanistic modeling framework to study the multilevel neural dynamics underlying neural information and cognitive processes.     

Evaluating the performance of artificial neural networks for the classification of freshwater benthic macroinvertebrates

Macroinvertebrates form an important functional component of aquatic ecosystems. Their ability to indicate various types of anthropogenic stressors is widely recognized which has made them an integral component of freshwater biomonitoring. The use of macroinvertebrates in biomonitoring is dependent on manual taxa identification which is currently a time-consuming and cost-intensive process conducted by highly trained taxonomical experts. Automated taxa identification of macroinvertebrates is a relatively recent research development. Previous studies have displayed great potential for solutions to this demanding data mining application. In this research we have a collection of 1350 images from eight different macroinvertebrate taxa and the aim is to examine the suitability of artificial neural networks (ANNs) for automated taxa identification of macroinvertebrates. More specifically, the focus is drawn on different training algorithms of Multi-Layer Perceptron (MLP), probabilistic neural network (PNN) and Radial Basis Function network (RBFN). We performed thorough experimental tests and we tested altogether 13 training algorithms for MLPs. The best classification accuracy of MLPs, 95.3%, was obtained by two conjugate gradient backpropagation variations and scaled conjugate gradient backpropagation. For PNN 92.8% and for RBFN 95.7% accuracies were achieved. The results show how important a proper choice of ANN is in order to obtain high accuracy in the automated taxa identification of macroinvertebrates and the obtained model can outperform the level of identification which is made by a taxonomist.     

New learning automata based algorithms for adaptation of backpropagation algorithm parameters

One popular learning algorithm for feedforward neural networks is the backpropagation (BP) algorithm which includes parameters, learning rate (eta), momentum factor (alpha) and steepness parameter (lambda). The appropriate selections of these parameters have large effects on the convergence of the algorithm. Many techniques that adaptively adjust these parameters have been developed to increase speed of convergence. In this paper, we shall present several classes of learning automata based solutions to the problem of adaptation of BP algorithm parameters. By interconnection of learning automata to the feedforward neural networks, we use learning automata scheme for adjusting the parameters eta, alpha, and lambda based on the observation of random response of the neural networks. One of the important aspects of the proposed schemes is its ability to escape from local minima with high possibility during the training period. The feasibility of proposed methods is shown through simulations on several problems.     

Modulation of the dynamics of cerebellar Purkinje cells through the interaction of excitatory and inhibitory feedforward pathways

The dynamics of cerebellar neuronal networks is controlled by the underlying building blocks of neurons and synapses between them. For which, the computation of Purkinje cells (PCs), the only output cells of the cerebellar cortex, is implemented through various types of neural pathways interactively routing excitation and inhibition converged to PCs. Such tuning of excitation and inhibition, coming from the gating of specific pathways as well as short-term plasticity (STP) of the synapses, plays a dominant role in controlling the PC dynamics in terms of firing rate and spike timing. PCs receive cascade feedforward inputs from two major neural pathways: the first one is the feedforward excitatory pathway from granule cells (GCs) to PCs; the second one is the feedforward inhibition pathway from GCs, via molecular layer interneurons (MLIs), to PCs. The GC-PC pathway, together with short-term dynamics of excitatory synapses, has been a focus over past decades, whereas recent experimental evidence shows that MLIs also greatly contribute to controlling PC activity. Therefore, it is expected that the diversity of excitation gated by STP of GC-PC synapses, modulated by strong inhibition from MLI-PC synapses, can promote the computation performed by PCs. However, it remains unclear how these two neural pathways are interacted to modulate PC dynamics. Here using a computational model of PC network installed with these two neural pathways, we addressed this question to investigate the change of PC firing dynamics at the level of single cell and network. We show that the nonlinear characteristics of excitatory STP dynamics can significantly modulate PC spiking dynamics mediated by inhibition. The changes in PC firing rate, firing phase, and temporal spike pattern, are strongly modulated by these two factors in different ways. MLIs mainly contribute to variable delays in the postsynaptic action potentials of PCs while modulated by excitation STP. Notably, the diversity of synchronization and pause response in the PC network is governed not only by the balance of excitation and inhibition, but also by the synaptic STP, depending on input burst patterns. Especially, the pause response shown in the PC network can only emerge with the interaction of both pathways. Together with other recent findings, our results show that the interaction of feedforward pathways of excitation and inhibition, incorporated with synaptic short-term dynamics, can dramatically regulate the PC activities that consequently change the network dynamics of the cerebellar circuit.     

Optimization of fed-batch bioreactor using neural network model

An algorithm using feedforward neural network model for determining optimal substrate feeding policies for fed-batch fermentation process is presented in this work. The algorithm involves developing the neural network model of the process using the sampled data. The trained neural network model in turn is used for optimization purposes. The advantages of this technique is that optimization can be achieved without detailed kinetic model of the process and the computation of gradient of objective function with respect to control variables is straightforward. The application of the technique is demonstrated with two examples, namely, production of secreted protein and invertase. The simulation results show that the discrete-time dynamics of fed-batch bioreactor can be satisfactorily approximated using a feedforward sigmoidal neural network. The optimal policies obtained with the neural network model agree reasonably well with the previously reported results.     

Recruitment learning of boolean functions in sparse random networks

This work presents a new class of neural network models constrained by biological levels of sparsity and weight-precision, and employing only local weight updates. Concept learning is accomplished through the rapid recruitment of existing network knowledge - complex knowledge being realised as a combination of existing basis concepts. Prior network knowledge is here obtained through the random generation of feedforward networks, with the resulting concept library tailored through distributional bias to suit a particular target class. Learning is exclusively local - through supervised Hebbian and Winnow updates - avoiding the necessity for backpropagation of error and allowing remarkably rapid learning. The approach is demonstrated upon concepts of varying difficulty, culminating in the well-known Monks and LED benchmark problems.