Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, e.g. fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of RNNs may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics.     

Implementing Gaussian process inference with neural networks

Gaussian processes compare favourably with backpropagation neural networks as a tool for regression, and Bayesian neural networks have Gaussian process behaviour when the number of hidden neurons tends to infinity. We describe a simple recurrent neural network with connection weights trained by one-shot Hebbian learning. This network amounts to a dynamical system which relaxes to a stable state in which it generates predictions identical to those of Gaussian process regression. In effect an infinite number of hidden units in a feed-forward architecture can be replaced by a merely finite number, together with recurrent connections.     

Sequential configuration model for firing patterns in local neural networks

This paper presents a sequential configuration model to represent the coordinated firing patterns of memory traces in groups of neurons in local networks. Computer simulations are used to study the dynamic properties of memory traces selectively retrieved from networks in which multiple memory traces have been embedded according to the sequential configuration model. Distinct memory traces which utilize the same neurons, but differ only in temporal sequencing are selectively retrievable. Firing patterns of constituent neurons of retrieved memory traces exhibit the main properties of neurons observed in multi microelectrode recordings. The paper shows how to adjust relative synaptic weightings so as to control the disruptive influences of cross-talk in multipy-embedded networks. The theoretical distinction between (primarily anatomical) beds and (primarily physiological) realizations underlines the fundamentally stochastic nature of network firing patterns, and allows the definition of 4 degrees of clarity of retrieved memory traces.     

Multi-scale inference of genetic trait architecture using biologically annotated neural networks

In this article, we present Biologically Annotated Neural Networks (BANNs), a nonlinear probabilistic framework for association mapping in genome-wide association (GWA) studies. BANNs are feedforward models with partially connected architectures that are based on biological annotations. This setup yields a fully interpretable neural network where the input layer encodes SNP-level effects, and the hidden layer models the aggregated effects among SNP-sets. We treat the weights and connections of the network as random variables with prior distributions that reflect how genetic effects manifest at different genomic scales. The BANNs software uses variational inference to provide posterior summaries which allow researchers to simultaneously perform (i) mapping with SNPs and (ii) enrichment analyses with SNP-sets on complex traits. Through simulations, we show that our method improves upon state-of-the-art association mapping and enrichment approaches across a wide range of genetic architectures. We then further illustrate the benefits of BANNs by analyzing real GWA data assayed in approximately 2,000 heterogenous stock of mice from the Wellcome Trust Centre for Human Genetics and approximately 7,000 individuals from the Framingham Heart Study. Lastly, using a random subset of individuals of European ancestry from the UK Biobank, we show that BANNs is able to replicate known associations in high and low-density lipoprotein cholesterol content.     

Multilayer neural networks and function reconstruction by using a priori knowledge

This paper proposes a new methodology to approximate functions by incorporating a priori information. The relationship between the proposed scheme and multilayer neural networks is explored theoretically and numerically. This approach is particularly interesting for the very relevant class of limited spectrum functions. The number of free parameters is smaller if compared to Back-Propagation Algorithm opening the way for better generalization results.     

Rule extraction from minimal neural networks for credit card screening

While feedforward neural networks have been widely accepted as effective tools for solving classification problems, the issue of finding the best network architecture remains unresolved, particularly so in real-world problem settings. We address this issue in the context of credit card screening, where it is important to not only find a neural network with good predictive performance but also one that facilitates a clear explanation of how it produces its predictions. We show that minimal neural networks with as few as one hidden unit provide good predictive accuracy, while having the added advantage of making it easier to generate concise and comprehensible classification rules for the user. To further reduce model size, a novel approach is suggested in which network connections from the input units to this hidden unit are removed by a very straightaway pruning procedure. In terms of predictive accuracy, both the minimized neural networks and the rule sets generated from them are shown to compare favorably with other neural network based classifiers. The rules generated from the minimized neural networks are concise and thus easier to validate in a real-life setting.     

A study on rule extraction from several combined neural networks

The problem of rule extraction from neural networks is NP-hard. This work presents a new technique to extract "if-then-else" rules from ensembles of DIMLP neural networks. Rules are extracted in polynomial time with respect to the dimensionality of the problem, the number of examples, and the size of the resulting network. Further, the degree of matching between extracted rules and neural network responses is 100%. Ensembles of DIMLP networks were trained on four data sets in the public domain. Extracted rules were on average significantly more accurate than those extracted from C4.5 decision trees.     

MetaNetter: inference and visualization of high-resolution metabolomic networks

SUMMARY: We present a Cytoscape plugin for the inference and visualization of networks from high-resolution mass spectrometry metabolomic data. The software also provides access to basic topological analysis. This open source, multi-platform software has been successfully used to interpret metabolomic experiments and will enable others using filtered, high mass accuracy mass spectrometric data sets to build and analyse networks. AVAILABILITY: http://compbio.dcs.gla.ac.uk/fabien/abinitio/abinitio.html     

Artificial neural networks: fundamentals, computing, design, and application

Artificial neural networks (ANNs) are relatively new computational tools that have found extensive utilization in solving many complex real-world problems. The attractiveness of ANNs comes from their remarkable information processing characteristics pertinent mainly to nonlinearity, high parallelism, fault and noise tolerance, and learning and generalization capabilities. This paper aims to familiarize the reader with ANN-based computing (neurocomputing) and to serve as a useful companion practical guide and toolkit for the ANNs modeler along the course of ANN project development. The history of the evolution of neurocomputing and its relation to the field of neurobiology is briefly discussed. ANNs are compared to both expert systems and statistical regression and their advantages and limitations are outlined. A bird's eye review of the various types of ANNs and the related learning rules is presented, with special emphasis on backpropagation (BP) ANNs theory and design. A generalized methodology for developing successful ANNs projects from conceptualization, to design, to implementation, is described. The most common problems that BPANNs developers face during training are summarized in conjunction with possible causes and remedies. Finally, as a practical application, BPANNs were used to model the microbial growth curves of S. flexneri. The developed model was reasonably accurate in simulating both training and test time-dependent growth curves as affected by temperature and pH.     

Codes, operations, measurements and neural networks

Numerous neural codes and primary neural operations (logical and arithmetical ones, mappings, transformations) were listed [e.g. Perkel, D., Bullock, T.H., 1968. Neurosci. Res. Program Bull 6, 221-348] during the past decades. None of them is ubiquitous or universal. In reality neural operations take place in continuous time and working with unreliable elements, but they still can be simulated with synchronized discrete time scales and chaotic models. Here, a possible neural mechanism, called 'measure like' code is introduced and examined. The neurons are regarded as measuring devices, dealing with 'measures', more or less in mathematical sense. The subadditivity--eminent property of measures--may be implemented with neuronal refractoriness and such synapses operate like particle counters with dead time. This hypothetical code is neither ubiquitous, nor universal, e.g. temporal summation (multiplication) causes just the opposite phenomenon, the supra-additivity also with respect to the number of spikes (anti-measures). This is a cause of more difficult neural implementation of OR gate, than that of the AND. Possibilities for transitional mechanisms (e.g. between traditional logical gates, etc.) are stressed here. Parameter tuning might change either code or operation.     

Algebraic comparison of metabolic networks, phylogenetic inference, and metabolic innovation

Background  

Comparison of metabolic networks is typically performed based on the organisms' enzyme contents. This approach disregards functional replacements as well as orthologies that are misannotated. Direct comparison of the structure of metabolic networks can circumvent these problems.     

NET-SYNTHESIS: a software for synthesis, inference and simplification of signal transduction networks

We present a software for combined synthesis, inference and simplification of signal transduction networks. The main idea of our method lies in representing observed indirect causal relationships as network paths and using techniques from combinatorial optimization to find the sparsest graph consistent with all experimental observations. We illustrate the biological usability of our software by applying it to a previously published signal transduction network and by using it to synthesize and simplify a novel network corresponding to activation-induced cell death in large granular lymphocyte leukemia. AVAILABILITY: NET-SYNTHESIS is freely downloadable from http://www.cs.uic.edu/~dasgupta/network-synthesis/     

Spiking networks for Bayesian inference and choice

Systems neuroscience traditionally conceptualizes a population of spiking neurons as merely encoding the value of a stimulus. Yet, psychophysics has revealed that people take into account stimulus uncertainty when performing sensory or motor computations and do so in a nearly Bayes-optimal way. This suggests that neural populations do not encode just a single value but an entire probability distribution over the stimulus. Several such probabilistic codes have been proposed, including one that utilizes the structure of neural variability to enable simple neural implementations of probabilistic computations such as optimal cue integration. This approach provides a quantitative link between Bayes-optimal behaviors and specific neural operations. It allows for novel ways to evaluate probabilistic codes and for predictions for physiological population recordings.     

Knowledge extraction from evolving spiking neural networks with rank order population coding

This paper demonstrates how knowledge can be extracted from evolving spiking neural networks with rank order population coding. Knowledge discovery is a very important feature of intelligent systems. Yet, a disproportionally small amount of research is centered on the issue of knowledge extraction from spiking neural networks which are considered to be the third generation of artificial neural networks. The lack of knowledge representation compatibility is becoming a major detriment to end users of these networks. We show that a high-level knowledge can be obtained from evolving spiking neural networks. More specifically, we propose a method for fuzzy rule extraction from an evolving spiking network with rank order population coding. The proposed method was used for knowledge discovery on two benchmark taste recognition problems where the knowledge learnt by an evolving spiking neural network was extracted in the form of zero-order Takagi-Sugeno fuzzy IF-THEN rules.