Minimal models of multidimensional computations

The multidimensional computations performed by many biological systems are often characterized with limited information about the correlations between inputs and outputs. Given this limitation, our approach is to construct the maximum noise entropy response function of the system, leading to a closed-form and minimally biased model consistent with a given set of constraints on the input/output moments; the result is equivalent to conditional random field models from machine learning. For systems with binary outputs, such as neurons encoding sensory stimuli, the maximum noise entropy models are logistic functions whose arguments depend on the constraints. A constraint on the average output turns the binary maximum noise entropy models into minimum mutual information models, allowing for the calculation of the information content of the constraints and an information theoretic characterization of the system's computations. We use this approach to analyze the nonlinear input/output functions in macaque retina and thalamus; although these systems have been previously shown to be responsive to two input dimensions, the functional form of the response function in this reduced space had not been unambiguously identified. A second order model based on the logistic function is found to be both necessary and sufficient to accurately describe the neural responses to naturalistic stimuli, accounting for an average of 93% of the mutual information with a small number of parameters. Thus, despite the fact that the stimulus is highly non-Gaussian, the vast majority of the information in the neural responses is related to first and second order correlations. Our results suggest a principled and unbiased way to model multidimensional computations and determine the statistics of the inputs that are being encoded in the outputs.     

Storage of correlated patterns in standard and bistable Purkinje cell models

The cerebellum has long been considered to undergo supervised learning, with climbing fibers acting as a 'teaching' or 'error' signal. Purkinje cells (PCs), the sole output of the cerebellar cortex, have been considered as analogs of perceptrons storing input/output associations. In support of this hypothesis, a recent study found that the distribution of synaptic weights of a perceptron at maximal capacity is in striking agreement with experimental data in adult rats. However, the calculation was performed using random uncorrelated inputs and outputs. This is a clearly unrealistic assumption since sensory inputs and motor outputs carry a substantial degree of temporal correlations. In this paper, we consider a binary output neuron with a large number of inputs, which is required to store associations between temporally correlated sequences of binary inputs and outputs, modelled as Markov chains. Storage capacity is found to increase with both input and output correlations, and diverges in the limit where both go to unity. We also investigate the capacity of a bistable output unit, since PCs have been shown to be bistable in some experimental conditions. Bistability is shown to enhance storage capacity whenever the output correlation is stronger than the input correlation. Distribution of synaptic weights at maximal capacity is shown to be independent on correlations, and is also unaffected by the presence of bistability.     

A memory-based self-generated basis function neural network.

The paper presents a novel memory-based Self-Generated Basis Function Neural Network (SGBFN) that is composed of small CMACs. The SGBFN requires much smaller memory space than the conventional CMAC and has an excellent learning convergence property compared to multilayer neural networks. Each CMAC in the new structure takes a subset of problem inputs as its inputs. Several CMACs that have different subsets of inputs form a submodule and a group of submodules form a neural network. The output of a submodule is the product of its CMACs' outputs. Each submodule implements a self-generated basis function, which is developed during the learning. The output of the neural network is the sum of the outputs from the submodules. Using only a subset of inputs in each CMAC significantly reduces the required memory space in high-dimensional modeling. With the same size of memory, the new structure is able to achieve a much smaller learning error compared to the conventional CMAC.     

A global competitive neural network

 A study is presented of a set of coupled nets proposed to function as a global competitive network. One net, of hidden nodes, is composed solely of inhibitory neurons and is excitatorily driven and feeds back in a disinhibitory manner to an input net which itself feeds excitatorily to a (cortical) output net. The manner in which the former input and hidden inhibitory net function so as to enhance outputs as compared with inputs, and the further enhancements when the cortical net is added, are explored both mathematically and by simulation. This is extended to learning on cortical afferent and lateral connections. A global wave structure, arising on the inhibitory net in a similar manner to that of pattern formation in a negative laplacian net, is seen to be important to all of these activities. Simulations are only performed in one dimension, although the global nature of the activity is expected to extend to higher dimensions. Possible implications are briefly discussed. Received: 21 November 1993/Accepted in revised form: 30 June 1994     

What the mantis shrimp's eye (possibly) tells its raptorial appendages

On the distal lamina surface of stomatoped eyes a matrix of horizontal and vertical nerve fibers processes the complex input patterns. The output of these integrating fibers is a spike discharge pattern correlated to the input pattern. Here we introduce a ± neuron which either subtracts or sums the outputs from the set of horizontal and vertical fibers activated at that moment. The output pattern of spike frequencies of the ± neuron locates a target in space. Over a parallel channel outputs from the vertical fibers pass a bandpass filter and constitute one of the inputs for a decision neuron. The other input derives from specialized ommatidia at the center of the eye. If both inputs arrive within a short time interval, they sum, the threshold is reached and the decision neuron fires. Spikes from the decision neuron are the final cue for the muscles of the raptorial appendages. The appendages shoot forward and hit the prey with high precision.     

An integrated approach to optimization of Escherichia coli fermentations using historical data

Using a fermentation database for Escherichia coli producing green fluorescent protein (GFP), we have implemented a novel three-step optimization method to identify the process input variables most important in modeling the fermentation, as well as the values of those critical input variables that result in an increase in the desired output. In the first step of this algorithm, we use either decision-tree analysis (DTA) or information theoretic subset selection (ITSS) as a database mining technique to identify which process input variables best classify each of the process outputs (maximum cell concentration, maximum product concentration, and productivity) monitored in the experimental fermentations. The second step of the optimization method is to train an artificial neural network (ANN) model of the process input-output data, using the critical inputs identified in the first step. Finally, a hybrid genetic algorithm (hybrid GA), which includes both gradient and stochastic search methods, is used to identify the maximum output modeled by the ANN and the values of the input conditions that result in that maximum. The results of the database mining techniques are compared, both in terms of the inputs selected and the subsequent ANN performance. For the E. coli process used in this study, we identified 6 inputs from the original 13 that resulted in an ANN that best modeled the GFP fluorescence outputs of an independent test set. Values of the six inputs that resulted in a modeled maximum fluorescence were identified by applying a hybrid GA to the ANN model developed. When these conditions were tested in laboratory fermentors, an actual maximum fluorescence of 2.16E6 AU was obtained. The previous high value of fluorescence that was observed was 1.51E6 AU. Thus, this input condition set that was suggested by implementing the proposed optimization scheme on the available historical database increased the maximum fluorescence by 55%.     

Dynamics of memory representations in networks with novelty-facilitated synaptic plasticity

The ability to associate some stimuli while differentiating between others is an essential characteristic of biological memory. Theoretical models identify memories as attractors of neural network activity, with learning based on Hebb-like synaptic modifications. Our analysis shows that when network inputs are correlated, this mechanism results in overassociations, even up to several memories "merging" into one. To counteract this tendency, we introduce a learning mechanism that involves novelty-facilitated modifications, accentuating synaptic changes proportionally to the difference between network input and stored memories. This mechanism introduces a dependency of synaptic modifications on previously acquired memories, enabling a wide spectrum of memory associations, ranging from absolute discrimination to complete merging. The model predicts that memory representations should be sensitive to learning order, consistent with recent psychophysical studies of face recognition and electrophysiological experiments on hippocampal place cells. The proposed mechanism is compatible with a recent biological model of novelty-facilitated learning in hippocampal circuitry.     

AmbiPack: A systematic algorithm for packing of macromolecular structures with ambiguous distance constraints

The determination of structures of multimers presents interesting new challenges. The structure(s) of the individual monomers must be found and the transformations to produce the packing interfaces must be described. A substantial difficulty results from ambiguities in assigning intermolecular distance measurements (from nuclear magnetic resonance, for example) to particular intermolecular interfaces in the structure. Here we present a rapid and efficient method to solve the packing and the assignment problems simultaneously given rigid monomer structures and (potentially ambiguous) intermolecular distance measurements. A promising application of this algorithm is to couple it with a monomer searching protocol such that each monomer structure consistent with intramolecular constraints can be subsequently input to the current algorithm to check whether it is consistent with (potentially ambiguous) intermolecular constraints. The algorithm AmbiPack uses a hierarchical division of the search space and the branch-and-bound algorithm to eliminate infeasible regions of the space. Local search methods are then focused on the remaining space. The algorithm generally runs faster as more constraints are included because more regions of the search space can be eliminated. This is not the case for other methods, for which additional constraints increase the complexity of the search space. The algorithm presented is guaranteed to find all solutions to a predetermined resolution. This resolution can be chosen arbitrarily to produce outputs at various level of detail. Illustrative applications are presented for the P22 tailspike protein (a trimer) and portions of β-amyloid (an ordered aggregate). Proteins 32:26–42, 1998. © 1998 Wiley-Liss, Inc.     

Conditioned reflex: detector and command neuron

Conditioned reflex is characterized by plasticity resulting in a bilateral selective input-output linking. In simple nervous systems, input stimuli are represented by selective detectors connected with command neurons through plastic synapses strengthened during associative learning and weakened during extinction. The process of associative learning is due to temporal coincidence of excitation in both detector and command neurons. Short-term memory within a plastic synapses is mediated by phosphorilation of postsynaptic receptor molecules not requiring protein synthesis. Long-term synaptic memory parallels expression of immediate early genes that mediates structural gene expression and protein synthesis. A simple detector-command neuron association becomes more complex in the course of evolution. Input mechanism is supplemented with predetector interneurons preceding detectors. Detector selectively tuned to specific input stimulus is converging on a command neuron constitute selectivity mechanism for conditioned reflexes to complex stimuli. The complication also concerns the output mechanisms. Command neurons become more specialized, and an additional link of premotor interneurons is incorporated between command neurons and motor neurons. Via synapses, the command neurons can produce excitation in a particular set of premotor neurons controlling a specific set of motor neurons responsible for behavioral act configuration. Specialization of command neurons in combination with premotor neuron structures increases the variability of outputs. Conditioned reflexes with more complex inputs and more flexible outputs determine the diversity of acquired behaviors.     

Just do it: action-dependent learning allows sensory prediction

Sensory-motor learning is commonly considered as a mapping process, whereby sensory information is transformed into the motor commands that drive actions. However, this directional mapping, from inputs to outputs, is part of a loop; sensory stimuli cause actions and vice versa. Here, we explore whether actions affect the understanding of the sensory input that they cause. Using a visuo-motor task in humans, we demonstrate two types of learning-related behavioral effects. Stimulus-dependent effects reflect stimulus-response learning, while action-dependent effects reflect a distinct learning component, allowing the brain to predict the forthcoming sensory outcome of actions. Together, the stimulus-dependent and the action-dependent learning components allow the brain to construct a complete internal representation of the sensory-motor loop.     

Learning shapes spontaneous activity itinerating over memorized states

Learning is a process that helps create neural dynamical systems so that an appropriate output pattern is generated for a given input. Often, such a memory is considered to be included in one of the attractors in neural dynamical systems, depending on the initial neural state specified by an input. Neither neural activities observed in the absence of inputs nor changes caused in the neural activity when an input is provided were studied extensively in the past. However, recent experimental studies have reported existence of structured spontaneous neural activity and its changes when an input is provided. With this background, we propose that memory recall occurs when the spontaneous neural activity changes to an appropriate output activity upon the application of an input, and this phenomenon is known as bifurcation in the dynamical systems theory. We introduce a reinforcement-learning-based layered neural network model with two synaptic time scales; in this network, I/O relations are successively memorized when the difference between the time scales is appropriate. After the learning process is complete, the neural dynamics are shaped so that it changes appropriately with each input. As the number of memorized patterns is increased, the generated spontaneous neural activity after learning shows itineration over the previously learned output patterns. This theoretical finding also shows remarkable agreement with recent experimental reports, where spontaneous neural activity in the visual cortex without stimuli itinerate over evoked patterns by previously applied signals. Our results suggest that itinerant spontaneous activity can be a natural outcome of successive learning of several patterns, and it facilitates bifurcation of the network when an input is provided.     

Feedforward neural network with adaptive reference pattern layer.

A hybrid neural network architecture is investigated for modeling purposes. The proposed hybrid is based on the multilayer perceptron (MLP) network. In addition to the usual hidden layers, the first hidden layer is selected to be an adaptive reference pattern layer. Each unit in this new layer incorporates a reference pattern that is located somewhere in the space spanned by the input variables. The outputs of these units are the component wise-squared differences between the elements of a reference pattern and the inputs. The reference pattern layer has some resemblance to the hidden layer of the radial basis function (RBF) networks. Therefore the proposed design can be regarded as a sort of hybrid of MLP and RBF networks. The presented benchmark experiments show that the proposed hybrid can provide significant advantages over standard MLPs and RBFs in terms of fast and efficient learning, and compact network structure.     

Computer simulations of synchrony and oscillations evoked by two coherent inputs

Coherent oscillations have been reported in multiple cortical areas. This study examines the characteristics of output spikes through computer simulations when the neural network model receives periodic/aperiodic spatiotemporal spikes with modulated/constant populational activity from two pathways. Synchronous oscillations which have the same period as the input are observed in response to periodic input patterns regardless of populational activity. The results confirm that the output frequency of synchrony is essentially determined by the period of the repeated input patterns. On the other hand, weak periodic outputs are observed when aperiodic spikes are input with modulated populational activity. In this case, higher firing rates are necessary to input for higher frequency oscillations. The spike-timing-dependent plasticity suppresses the spikes which do not contribute to the synchrony for periodic inputs. This effect corresponds to the experimental reports that learning sharpens the synchrony in the motor cortex. These results suggest that spatiotemporal spike patterns should be entrained on modulated populational activity to transmit oscillatory information effectively in the convergent pathway.     

Combined effects of LTP/LTD and synaptic scaling in formation of discrete and line attractors with persistent activity from non-trivial baseline

In this article, we analyze combined effects of LTP/LTD and synaptic scaling and study the creation of persistent activity from a periodic or chaotic baseline attractor. The bifurcations leading to the creation of new attractors have been detailed; this was achieved using a mean field approximation. Attractors encoding persistent activity can notably appear via generalized period-doubling bifurcations, tangent bifurcations of the second iterates or boundary crises, after which the basins of attraction become irregular. Synaptic scaling is shown to maintain the coexistence of a state of persistent activity and the baseline. According to the rate of change of the external inputs, different types of attractors can be formed: line attractors for rapidly changing external inputs and discrete attractors for constant external inputs.     

Spike Triggered Covariance in Strongly Correlated Gaussian Stimuli

Many biological systems perform computations on inputs that have very large dimensionality. Determining the relevant input combinations for a particular computation is often key to understanding its function. A common way to find the relevant input dimensions is to examine the difference in variance between the input distribution and the distribution of inputs associated with certain outputs. In systems neuroscience, the corresponding method is known as spike-triggered covariance (STC). This method has been highly successful in characterizing relevant input dimensions for neurons in a variety of sensory systems. So far, most studies used the STC method with weakly correlated Gaussian inputs. However, it is also important to use this method with inputs that have long range correlations typical of the natural sensory environment. In such cases, the stimulus covariance matrix has one (or more) outstanding eigenvalues that cannot be easily equalized because of sampling variability. Such outstanding modes interfere with analyses of statistical significance of candidate input dimensions that modulate neuronal outputs. In many cases, these modes obscure the significant dimensions. We show that the sensitivity of the STC method in the regime of strongly correlated inputs can be improved by an order of magnitude or more. This can be done by evaluating the significance of dimensions in the subspace orthogonal to the outstanding mode(s). Analyzing the responses of retinal ganglion cells probed with Gaussian noise, we find that taking into account outstanding modes is crucial for recovering relevant input dimensions for these neurons.     

Reconstructing muscle activation during normal walking: a comparison of symbolic and connectionist machine learning techniques

One symbolic (rule-based inductive learning) and one connectionist (neural network) machine learning technique were used to reconstruct muscle activation patterns from kinematic data measured during normal human walking at several speeds. The activation patterns (or desired outputs) consisted of surface electromyographic (EMG) signals from the semitendinosus and vastus medialis muscles. The inputs consisted of flexion and extension angles measured at the hip and knee of the ipsilateral leg, their first and second derivatives, and bilateral foot contact information. The training set consisted of data from six trials, at two different speeds. The testing set consisted of data from two additional trials (one at each speed), which were not in the training set. It was possible to reconstruct the muscular activation at both speeds using both techniques. Timing of the reconstructed signals was accurate. The integrated value of the activation bursts was less accurate. The neural network gave a continuous output, whereas the rule-based inductive learning rule tree gave a quantised activation level. The advantage of rule-based inductive learning was that the rules used were both explicit and comprehensible, whilst the rules used by the neural network were implicit within its structure and not easily comprehended. The neural network was able to reconstruct the activation patterns of both muscles from one network, whereas two separate rule sets were needed for the rule-based technique. It is concluded that machine learning techniques, in comparison to explicit inverse muscular skeletal models, show good promise in modelling nearly cyclic movements such as locomotion at varying walking speeds. However, they do not provide insight into the biomechanics of the system, because they are not based on the biomechanical structure of the system.     

Adaptive combinatorial design to explore large experimental spaces: approach and validation

Systems biology requires mathematical tools not only to analyse large genomic datasets, but also to explore large experimental spaces in a systematic yet economical way. We demonstrate that two-factor combinatorial design (CD), shown to be useful in software testing, can be used to design a small set of experiments that would allow biologists to explore larger experimental spaces. Further, the results of an initial set of experiments can be used to seed further 'Adaptive' CD experimental designs. As a proof of principle, we demonstrate the usefulness of this Adaptive CD approach by analysing data from the effects of six binary inputs on the regulation of genes in the N-assimilation pathway of Arabidopsis. This CD approach identified the more important regulatory signals previously discovered by traditional experiments using far fewer experiments, and also identified examples of input interactions previously unknown. Tests using simulated data show that Adaptive CD suffers from fewer false positives than traditional experimental designs in determining decisive inputs, and succeeds far more often than traditional or random experimental designs in determining when genes are regulated by input interactions. We conclude that Adaptive CD offers an economical framework for discovering dominant inputs and interactions that affect different aspects of genomic outputs and organismal responses.     

厦门市生态经济系统物质流分析

运用物质流分析(MFA)方法和STIRPAT模型,对1996～2007年厦门生态经济系统物质输入与输出进行分析,结果表明:(1)在不考虑水的情况下,物质输入与输出不断增加(年均增长率分别为11.48%、11.41%),但均小于GDP增长速度(15.94%),二者成正比;物质流增长集中表现在对金属、非金属矿物的需求及化石燃料燃烧废气、工业废气的排放.(2)用水量和废水排放量均不断增加,尤以生活污水排放量增长速度较快,加重了区域环境的压力.(3)物质输入与GDP、物质输出与GDP呈良好线性关系.厦门经济发展很大程度上依赖资源消耗.(4)单位GDP物质输入与输出均不断减小,表明资源利用率、处置率明显提高,区域逐步走向生态环境与社会经济的协调发展.(5)构建了厦门物质输入驱动机制的STIRPAT模型,得出人口数量、富裕程度、技术水平或经济结构每分别发生1%的变化,将引起输入量相应发生0.99%、0.98%、0.17%、0.31%的变化.提升技术水平和优化经济结构具有较大调控空间,将是厦门物质减量化战略的实施重点.     

Axon shape as a basis for multinode functional units in a hierarchical neural model

The ability of animals to perform fixed action patterns and to access information by categories suggests that there are several types of hierarchical organization in the nervous system. This paper employs data about axon shape and neurotransmitter effect to demonstrate the emergence of hierarchical structure in a neural model. Two dimensions of neural classification, axon shape and neurotransmitter effect, are used to generate a five-node-type neural model. Neurons are classified as interneurons, relay cells, and monoamine transmitters on the basis of axon shape; the transmitter classifications include excitatory, inhibitory, and parameter-changing. The five types of nodes in the model correspond to all the biologically observed combinations: excitatory and inhibitory short-range, excitatory and inhibitory long-range-directional, and long-lasting long-range-diffuse nodes. The emergence of multinode functional units (MFUs) from the five-node-type model is mathematically demonstrated. These units correspond to cortical columns anatomically defined by the axon fields of relay cells, and are called columnar multinode functional units (CMFUs). CMFUs may, in turn, be part of larger functional groups designated coherent populations, which consist of widely distributed CMFUs in retinotopically equivalent locations. The existence of coherent populations imposes a three-level hierarchical structure on the model. To represent this hierarchical structure, a new type of CMFU node, which has a set of vector-valued inputs and outputs, is introduced. Each CMFU node contains a system of short-range nodes which supplies it with vector-valued inputs. Sets of long-range-diffuse nodes are also treated as vector-valued nodes whose outputs control the size and number of coherent populations. The role of coherent populations and hierarchical organization in the nervous system is discussed for such cognitive tasks as visual perception, attention and learning. Physiological and behavioral evidence are cited which support the existence of a similar three-level hierarchy in vertebrate brains.     

An innovative approach for testing bioinformatics programs using metamorphic testing

Background  

Recent advances in experimental and computational technologies have fueled the development of many sophisticated bioinformatics programs. The correctness of such programs is crucial as incorrectly computed results may lead to wrong biological conclusion or misguide downstream experimentation. Common software testing procedures involve executing the target program with a set of test inputs and then verifying the correctness of the test outputs. However, due to the complexity of many bioinformatics programs, it is often difficult to verify the correctness of the test outputs. Therefore our ability to perform systematic software testing is greatly hindered.