Software reliability prediction using recurrent neural network with Bayesian regularization

A recurrent neural network modeling approach for software reliability prediction with respect to cumulative failure time is proposed. Our proposed network structure has the capability of learning and recognizing the inherent internal temporal property of cumulative failure time sequence. Further, by adding a penalty term of sum of network connection weights, Bayesian regularization is applied to our network training scheme to improve the generalization capability and lower the susceptibility of overfitting. The performance of our proposed approach has been tested using four real-time control and flight dynamic application data sets. Numerical results show that our proposed approach is robust across different software projects, and has a better performance with respect to both goodness-of-fit and next-step-predictability compared to existing neural network models for failure time prediction.     

Theoretical models of DNA topology simplification by type IIA DNA topoisomerases

It was discovered 12 years ago that type IIA topoisomerases can simplify DNA topology—the steady-state fractions of knots and links created by the enzymes are many times lower than the corresponding equilibrium fractions. Though this property of the enzymes made clear biological sense, it was not clear how small enzymes could selectively change the topology of very large DNA molecules, since topology is a global property and cannot be determined by a local DNA–protein interaction. A few models, suggested to explain the phenomenon, are analyzed in this review. We also consider experimental data that both support and contravene these models.     

A novel approach for short-term load forecasting using support vector machines

A support vector machine (SVM) modeling approach for short-term load forecasting is proposed. The SVM learning scheme is applied to the power load data, forcing the network to learn the inherent internal temporal property of power load sequence. We also study the performance when other related input variables such as temperature and humidity are considered. The performance of our proposed SVM modeling approach has been tested and compared with feed-forward neural network and cosine radial basis function neural network approaches. Numerical results show that the SVM approach yields better generalization capability and lower prediction error compared to those neural network approaches.     

Perspective: network-guided pattern formation of neural dynamics

The understanding of neural activity patterns is fundamentally linked to an understanding of how the brain''s network architecture shapes dynamical processes. Established approaches rely mostly on deviations of a given network from certain classes of random graphs. Hypotheses about the supposed role of prominent topological features (for instance, the roles of modularity, network motifs or hierarchical network organization) are derived from these deviations. An alternative strategy could be to study deviations of network architectures from regular graphs (rings and lattices) and consider the implications of such deviations for self-organized dynamic patterns on the network. Following this strategy, we draw on the theory of spatio-temporal pattern formation and propose a novel perspective for analysing dynamics on networks, by evaluating how the self-organized dynamics are confined by network architecture to a small set of permissible collective states. In particular, we discuss the role of prominent topological features of brain connectivity, such as hubs, modules and hierarchy, in shaping activity patterns. We illustrate the notion of network-guided pattern formation with numerical simulations and outline how it can facilitate the understanding of neural dynamics.     

How robust are neural network models of stimulus generalization?

Artificial feed-forward neural networks are commonly used as a tool for modelling stimulus selection and animal signalling. A key finding of stimulus selection research has been generalization: if a given behaviour has been established to one stimulus, perceptually similar novel stimuli are likely to induce a similar response. Stimulus generalization, in feed-forward neural networks, automatically arises as a property of the network. This network property raises understandable concern regarding the sensitivity of the network to variation in its internal parameter values used in relation to its structure and to its training process. Researchers must have confidence that the predictions of their model follow from the underlying biology that they deliberately incorporated in the model, and not from often arbitrary choices about model implementation. We study how network training and parameter perturbations influence the qualitative and quantitative behaviour of a simple but general network. Specifically, for models of stimulus control we study the effect that parameter variation has on the shape of the generalization curves produced by the network. We show that certain network and training conditions produce undesirable artifacts that need to be avoided (or at least understood) when modelling stimulus selection.     

Artificial neural networks and the study of evolution of prey coloration

In this paper, I investigate the use of artificial neural networks in the study of prey coloration. I briefly review the anti-predator functions of prey coloration and describe both in general terms and with help of two studies as specific examples the use of neural network models in the research on prey coloration. The first example investigates the effect of visual complexity of background on evolution of camouflage. The second example deals with the evolutionary choice of defence strategy, crypsis or aposematism. I conclude that visual information processing by predators is central in evolution of prey coloration. Therefore, the capability to process patterns as well as to imitate aspects of predator's information processing and responses to visual information makes neural networks a well-suited modelling approach for the study of prey coloration. In addition, their suitability for evolutionary simulations is an advantage when complex or dynamic interactions are modelled. Since not all behaviours of neural network models are necessarily biologically relevant, it is important to validate a neural network model with empirical data. Bringing together knowledge about neural networks with knowledge about topics of prey coloration would provide a potential way to deepen our understanding of the specific appearances of prey coloration.     

Evolving Robust Gene Regulatory Networks

Design and implementation of robust network modules is essential for construction of complex biological systems through hierarchical assembly of ‘parts’ and ‘devices’. The robustness of gene regulatory networks (GRNs) is ascribed chiefly to the underlying topology. The automatic designing capability of GRN topology that can exhibit robust behavior can dramatically change the current practice in synthetic biology. A recent study shows that Darwinian evolution can gradually develop higher topological robustness. Subsequently, this work presents an evolutionary algorithm that simulates natural evolution in silico, for identifying network topologies that are robust to perturbations. We present a Monte Carlo based method for quantifying topological robustness and designed a fitness approximation approach for efficient calculation of topological robustness which is computationally very intensive. The proposed framework was verified using two classic GRN behaviors: oscillation and bistability, although the framework is generalized for evolving other types of responses. The algorithm identified robust GRN architectures which were verified using different analysis and comparison. Analysis of the results also shed light on the relationship among robustness, cooperativity and complexity. This study also shows that nature has already evolved very robust architectures for its crucial systems; hence simulation of this natural process can be very valuable for designing robust biological systems.     

A gain-field encoding of limb position and velocity in the internal model of arm dynamics

Adaptability of reaching movements depends on a computation in the brain that transforms sensory cues, such as those that indicate the position and velocity of the arm, into motor commands. Theoretical consideration shows that the encoding properties of neural elements implementing this transformation dictate how errors should generalize from one limb position and velocity to another. To estimate how sensory cues are encoded by these neural elements, we designed experiments that quantified spatial generalization in environments where forces depended on both position and velocity of the limb. The patterns of error generalization suggest that the neural elements that compute the transformation encode limb position and velocity in intrinsic coordinates via a gain-field; i.e., the elements have directionally dependent tuning that is modulated monotonically with limb position. The gain-field encoding makes the counterintuitive prediction of hypergeneralization: there should be growing extrapolation beyond the trained workspace. Furthermore, nonmonotonic force patterns should be more difficult to learn than monotonic ones. We confirmed these predictions experimentally.     

What can we teach Lymnaea and what can Lymnaea teach us?

This review describes the advantages of adopting a molluscan complementary model, the freshwater snail Lymnaea stagnalis, to study the neural basis of learning and memory in appetitive and avoidance classical conditioning; as well as operant conditioning of its aerial respiratory and escape behaviour. We firstly explored ‘what we can teach Lymnaea’ by discussing a variety of sensitive, solid, easily reproducible and simple behavioural tests that have been used to uncover the memory abilities of this model system. Answering this question will allow us to open new frontiers in neuroscience and behavioural research to enhance our understanding of how the nervous system mediates learning and memory. In fact, from a translational perspective, Lymnaea and its nervous system can help to understand the neural transformation pathways from behavioural output to sensory coding in more complex systems like the mammalian brain. Moving on to the second question: ‘what can Lymnaea teach us?’, it is now known that Lymnaea shares important associative learning characteristics with vertebrates, including stimulus generalization, generalization of extinction and discriminative learning, opening the possibility to use snails as animal models for neuroscience translational research.     

Self-organization of associative memory and pattern classification: recurrent signal processing on topological feature maps

We extend the neural concepts of topological feature maps towards self-organization of auto-associative memory and hierarchical pattern classification. As is well-known, topological maps for statistical data sets store information on the associated probability densities. To extract that information we introduce a recurrent dynamics of signal processing. We show that the dynamics converts a topological map into an auto-associative memory for real-valued feature vectors which is capable to perform a cluster analysis. The neural network scheme thus developed represents a generalization of non-linear matrix-type associative memories. The results naturally lead to the concept of a feature atlas and an associated scheme of self-organized, hierarchical pattern classification.     

A biologically inspired neural network for dynamic programming

An artificial neural network with a two-layer feedback topology and generalized recurrent neurons, for solving nonlinear discrete dynamic optimization problems, is developed. A direct method to assign the weights of neural networks is presented. The method is based on Bellmann's Optimality Principle and on the interchange of information which occurs during the synaptic chemical processing among neurons. The neural network based algorithm is an advantageous approach for dynamic programming due to the inherent parallelism of the neural networks; further it reduces the severity of computational problems that can occur in methods like conventional methods. Some illustrative application examples are presented to show how this approach works out including the shortest path and fuzzy decision making problems.     

On the rate of convergence in topology preserving neural networks

A formal analysis of the neighborhood interaction function selection in the topology preserving unsupervised neural network is presented in this paper. The definition of the neighborhood interaction function is motivated by anatomical evidence as opposed to what is currently used, which is a uniform neighborhood interaction set. By selecting a neighborhood interaction function with a neighborhood amplitude of interaction which is decreasing in spatial domain the topological order is always enforced and the rate of self-organization to final equilibrium state is improved. Several simulations are carried out to show the improvement in rate between using a neighborhood interaction function vs. using a neighborhood interaction set. An error measure functional is further defined to compare the two approaches quantitatively.     

Brain rhythms reveal a hierarchical network organization

Recordings of ongoing neural activity with EEG and MEG exhibit oscillations of specific frequencies over a non-oscillatory background. The oscillations appear in the power spectrum as a collection of frequency bands that are evenly spaced on a logarithmic scale, thereby preventing mutual entrainment and cross-talk. Over the last few years, experimental, computational and theoretical studies have made substantial progress on our understanding of the biophysical mechanisms underlying the generation of network oscillations and their interactions, with emphasis on the role of neuronal synchronization. In this paper we ask a very different question. Rather than investigating how brain rhythms emerge, or whether they are necessary for neural function, we focus on what they tell us about functional brain connectivity. We hypothesized that if we were able to construct abstract networks, or "virtual brains", whose dynamics were similar to EEG/MEG recordings, those networks would share structural features among themselves, and also with real brains. Applying mathematical techniques for inverse problems, we have reverse-engineered network architectures that generate characteristic dynamics of actual brains, including spindles and sharp waves, which appear in the power spectrum as frequency bands superimposed on a non-oscillatory background dominated by low frequencies. We show that all reconstructed networks display similar topological features (e.g. structural motifs) and dynamics. We have also reverse-engineered putative diseased brains (epileptic and schizophrenic), in which the oscillatory activity is altered in different ways, as reported in clinical studies. These reconstructed networks show consistent alterations of functional connectivity and dynamics. In particular, we show that the complexity of the network, quantified as proposed by Tononi, Sporns and Edelman, is a good indicator of brain fitness, since virtual brains modeling diseased states display lower complexity than virtual brains modeling normal neural function. We finally discuss the implications of our results for the neurobiology of health and disease.     

Evaluating machine learning approaches for aiding probe selection for gene-expression arrays

MOTIVATION: Microarrays are a fast and cost-effective method of performing thousands of DNA hybridization experiments simultaneously. DNA probes are typically used to measure the expression level of specific genes. Because probes greatly vary in the quality of their hybridizations, choosing good probes is a difficult task. If one could accurately choose probes that are likely to hybridize well, then fewer probes would be needed to represent each gene in a gene-expression microarray, and, hence, more genes could be placed on an array of a given physical size. Our goal is to empirically evaluate how successfully three standard machine-learning algorithms-na?ve Bayes, decision trees, and artificial neural networks-can be applied to the task of predicting good probes. Fortunately it is relatively easy to get training examples for such a learning task: place various probes on a gene chip, add a sample where the corresponding genes are highly expressed, and then record how well each probe measures the presence of its corresponding gene. With such training examples, it is possible that an accurate predictor of probe quality can be learned. RESULTS: Two of the learning algorithms we investigate-na?ve Bayes and neural networks-learn to predict probe quality surprisingly well. For example, in the top ten predicted probes for a given gene not used for training, on average about five rank in the top 2.5% of that gene's hundreds of possible probes. Decision-tree induction and the simple approach of using predicted melting temperature to rank probes perform significantly worse than these two algorithms. The features we use to represent probes are very easily computed and the time taken to score each candidate probe after training is minor. Training the na?ve Bayes algorithm takes very little time, and while it takes over 10 times as long to train a neural network, that time is still not very substantial (on the order of a few hours on a desktop workstation). We also report the information contained in the features we use to describe the probes. We find the fraction of cytosine in the probe to be the most informative feature. We also find, not surprisingly, that the nucleotides in the middle of the probes sequence are more informative than those at the ends of the sequence.     

Neural network analysis of follow-up data in primary breast cancer

This paper reports on the performance of a recently developed neural network environment incorporating likelihood-based optimization and complexity reduction techniques in the analysis of breast cancer follow-up data with the goal of building up a clinical decision support system. The inputs to the neural network include classical factors such as grading, age, tumor size, estrogen and progesterone receptor measurements, as well as tumor biological markers such as PAI-1 and uPA. The network learns the structural relationship between these factors and the follow-up data. Examples of neural models for relapse-free survival are presented, which are based on data from 784 breast cancer patients who received their primary therapy at the Department of Obstetrics and Gynecology, Technische Universit?t München, Germany. The performance of the neural analysis as quantified by various indicators (likelihood, Kaplan-Meier curves, log-rank tests) was very high. For example, dividing the patients into two equally sized groups based on the neural score (i.e., cutoff = median score) leads to an estimated difference in relapse-free survival of 40% or better (80% vs. 40%) after 10 years in Kaplan-Meier analysis. Evidence for factor interactions as well as for time-varying impacts is presented. The neural network weights included in the models are significant at the 5% level. The use of neural network analysis and scoring in combination with strong tumor biological factors such as uPA and PAI-1 appears to result in a very effective risk group discrimination. Considerable additional comparison of data from different patient series will be required to establish the generalization capability more firmly. Nonetheless, the improvement of risk group discrimination represents an important step toward the use of neural networks for decision support in a clinical framework and in making the most of biological markers.     

Spatial confusion or clarity? Reply to Ricotta & Avena

Abstract. This communication is a response to a recent suggestion that ordinal data of the Braun‐Blanquet type (BB) can be directly or, after conversion to ranks, indirectly analysed by metric methods. I show that the proposals on structure in topological spaces made by Ricotta & Avena in a recent contribution to this Forum are confounding because (1) they use the term ‘topological’ in the inappropriate way, and (2) the measure they propose is in fact a metric, rather than merely topological. In addition, I illustrate with a few examples how a truly topological measure functions, so that the reader can appreciate the ideas behind their definitions. By reference to axiomatic measurement theory, I argue that whenever vegetation scientists know exactly at the outset what attributes they wish to express by relevé data, what questions they are asking and whenever they are aware of the basic properties of the BB scale, ordinal data analysis is still the most logical choice.     

A robust neural networks approach for spatial and intensity-dependent normalization of cDNA microarray data

MOTIVATION: Microarray experiments are affected by numerous sources of non-biological variation that contribute systematic bias to the resulting data. In a dual-label (two-color) cDNA or long-oligonucleotide microarray, these systematic biases are often manifested as an imbalance of measured fluorescent intensities corresponding to Sample A versus those corresponding to Sample B. Systematic biases also affect between-slide comparisons. Making effective corrections for these systematic biases is a requisite for detecting the underlying biological variation between samples. Effective data normalization is therefore an essential step in the confident identification of biologically relevant differences in gene expression profiles. Several normalization methods for the correction of systemic bias have been described. While many of these methods have addressed intensity-dependent bias, few have addressed both intensity-dependent and spatiality-dependent bias. RESULTS: We present a neural network-based normalization method for correcting the intensity- and spatiality-dependent bias in cDNA microarray datasets. In this normalization method, the dependence of the log-intensity ratio (M) on the average log-intensity (A) as well as on the spatial coordinates (X,Y) of spots is approximated with a feed-forward neural network function. Resistance to outliers is provided by assigning weights to each spot based on how distant their M values is from the median over the spots whose A values are similar, as well as by using pseudospatial coordinates instead of spot row and column indices. A comparison of the robust neural network method with other published methods demonstrates its potential in reducing both intensity-dependent bias and spatial-dependent bias, which translates to more reliable identification of truly regulated genes.     

Interplay between Graph Topology and Correlations of Third Order in Spiking Neuronal Networks

The study of processes evolving on networks has recently become a very popular research field, not only because of the rich mathematical theory that underpins it, but also because of its many possible applications, a number of them in the field of biology. Indeed, molecular signaling pathways, gene regulation, predator-prey interactions and the communication between neurons in the brain can be seen as examples of networks with complex dynamics. The properties of such dynamics depend largely on the topology of the underlying network graph. In this work, we want to answer the following question: Knowing network connectivity, what can be said about the level of third-order correlations that will characterize the network dynamics? We consider a linear point process as a model for pulse-coded, or spiking activity in a neuronal network. Using recent results from theory of such processes, we study third-order correlations between spike trains in such a system and explain which features of the network graph (i.e. which topological motifs) are responsible for their emergence. Comparing two different models of network topology—random networks of Erdős-Rényi type and networks with highly interconnected hubs—we find that, in random networks, the average measure of third-order correlations does not depend on the local connectivity properties, but rather on global parameters, such as the connection probability. This, however, ceases to be the case in networks with a geometric out-degree distribution, where topological specificities have a strong impact on average correlations.     

Adaptive control for mimo uncertain nonlinear systems using recurrent wavelet neural network

Recurrent wavelet neural network (RWNN) has the advantages such as fast learning property, good generalization capability and information storing ability. With these advantages, this paper proposes an RWNN-based adaptive control (RBAC) system for multi-input multi-output (MIMO) uncertain nonlinear systems. The RBAC system is composed of a neural controller and a bounding compensator. The neural controller uses an RWNN to online mimic an ideal controller, and the bounding compensator can provide smooth and chattering-free stability compensation. From the Lyapunov stability analysis, it is shown that all signals in the closed-loop RBAC system are uniformly ultimately bounded. Finally, the proposed RBAC system is applied to the MIMO uncertain nonlinear systems such as a mass-spring-damper mechanical system and a two-link robotic manipulator system. Simulation results verify that the proposed RBAC system can achieve favorable tracking performance with desired robustness without any chattering phenomenon in the control effort.