On problem solving with Hopfield neural networks

Hopfield and Tank have shown that neural networks can be used to solve certain computationally hard problems, in particular they studied the Traveling Salesman Problem (TSP). Based on network simulation results they conclude that analog VLSI neural nets can be promising in solving these problems. Recently, Wilson and Pawley presented the results of their simulations which contradict the original results and cast doubts on the usefulness of neural nets. In this paper we give the results of our simulations that clarify some of the discrepancies. We also investigate the scaling of TSP solutions found by neural nets as the size of the problem increases. Further, we consider the neural net solution of the Clustering Problem, also a computationally hard problem, and discuss the types of problems that appear to be well suited for a neural net approach.     

Improvements in protein secondary structure prediction by an enhanced neural network

Computational neural networks have recently been used to predict the mapping between protein sequence and secondary structure. They have proven adequate for determining the first-order dependence between these two sets, but have, until now, been unable to garner higher-order information that helps determine secondary structure. By adding neural network units that detect periodicities in the input sequence, we have modestly increased the secondary structure prediction accuracy. The use of tertiary structural class causes a marked increase in accuracy. The best case prediction was 79% for the class of all-alpha proteins. A scheme for employing neural networks to validate and refine structural hypotheses is proposed. The operational difficulties of applying a learning algorithm to a dataset where sequence heterogeneity is under-represented and where local and global effects are inadequately partitioned are discussed.     

Causal pattern recovery from neural spike train data using the Snap Shot Score

We present a new approach to learning directed information flow networks from multi-channel spike train data. A novel scoring function, the Snap Shot Score, is used to assess potential networks with respect to their quality of causal explanation for the data. Additionally, we suggest a generic concept of plausibility in order to assess network learning techniques under partial observability conditions. Examples demonstrate the assessment of networks with the Snap Shot Score, and neural network simulations show its performance in complex situations with partial observability. We discuss the application of the new score to real data and indicate how it can be modified to suit other neural data types.     

Theory of interface: Category theory,directed networks and evolution of biological networks

Biological networks have two modes. The first mode is static: a network is a passage on which something flows. The second mode is dynamic: a network is a pattern constructed by gluing functions of entities constituting the network. In this paper, first we discuss that these two modes can be associated with the category theoretic duality (adjunction) and derive a natural network structure (a path notion) for each mode by appealing to the category theoretic universality. The path notion corresponding to the static mode is just the usual directed path. The path notion for the dynamic mode is called lateral path which is the alternating path considered on the set of arcs. Their general functionalities in a network are transport and coherence, respectively. Second, we introduce a betweenness centrality of arcs for each mode and see how the two modes are embedded in various real biological network data. We find that there is a trade-off relationship between the two centralities: if the value of one is large then the value of the other is small. This can be seen as a kind of division of labor in a network into transport on the network and coherence of the network. Finally, we propose an optimization model of networks based on a quality function involving intensities of the two modes in order to see how networks with the above trade-off relationship can emerge through evolution. We show that the trade-off relationship can be observed in the evolved networks only when the dynamic mode is dominant in the quality function by numerical simulations. We also show that the evolved networks have features qualitatively similar to real biological networks by standard complex network analysis.     

A genetic algorithm with adaptive mutations and family competition for training neural networks

In this paper, we present a new evolutionary technique to train three general neural networks. Based on family competition principles and adaptive rules, the proposed approach integrates decreasing-based mutations and self-adaptive mutations to collaborate with each other. Different mutations act as global and local strategies respectively to balance the trade-off between solution quality and convergence speed. Our algorithm is then applied to three different task domains: Boolean functions, regular language recognition, and artificial ant problems. Experimental results indicate that the proposed algorithm is very competitive with comparable evolutionary algorithms. We also discuss the search power of our proposed approach.     

Interactions between neural networks: a mechanism for tuning chaos and oscillations

We show that chaos and oscillations in a higher-order binary neural network can be tuned effectively using interactions between neural networks. Our results suggest that network interactions may be useful as a means of adjusting the level of dynamic activities in systems that employ chaos and oscillations for information processing, or as a means of suppressing oscillatory behaviors in systems that require stability. URL: http:// www.ntu.edu.sg/home/elpwang     

Neural network models for promoter recognition

The problem of recognition of promoter sites in the DNA sequence has been treated with models of learning neural networks. The maximum network capacity admissible for this problem has been estimated on the basis of the total of experimental data available on the determined promoter sequences. The model of a block neural network has been constructed to satisfy this estimate and rules have been elaborated for its learning and testing. The learning process involves a small (of the order of 10%) part of the total set of promoter sequences. During this procedure the neural network develops a system of distinctive features (key words) to be used as a reference in identifying promoters against the background of random sequences. The learning quality is then tested with the whole set. The efficiency of promoter recognition has been found to amount to 94 to 99%. The probability of an arbitrary sequence being identified as a promoter is 2 to 6%.     

多比例时滞细胞神经网络的指数周期性与稳定性

研究了一类具多比例时滞细胞神经网络的全局指数周期性与稳定性.通过变换y(t)=x(e~t)将具多比例时滞的细胞神经网络变换成具常时滞变系数的细胞神经网络,利用一些分析技巧与构造合适的Lyapunov泛函,得到系统的周期解存在唯一且全局指数周期的时滞依赖的充分条件,判断方法简单易验证.并给出了两个例子及其数值仿真结果以支持所得结论.     

On the pressure and flow-rate distributions in tree-like and arterial-venous networks

A solution algorithm yielding the pressure and flow-rate distributions for steady flow in an arbitrary, tree-like network is provided. Given the tree topology, the conductance of each segment and the pressure distribution at the boundary nodes, the solution is obtained from a simple recursion based on perfect Gauss elimination. An iterative solution method using this algorithm is suggested to solve for the pressure and flow-rate distributions in an arbitrary diverging-converging (arterial-venous) network consisting of two tree-like networks which are connected to each other at the capillary nodes. A number of special solutions for tree-like networks are obtained for which the general algorithm is either simplified or can be replaced by closed form solutions of the pressure and flow-rate distributions. These special solutions can also be obtained for each tree of diverging-converging networks having particular topologies and conductance distributions. Sample numerical results are provided.     

Neural Network Models for Promoter Recognition

Abstract

The problem of recognition of promoter sites in the DNA sequence has been treated with models of learning neural networks. The maximum network capacity admissible for this problem has been estimated on the basis of the total of experimental data available on the determined promoter sequences. The model of a block neural network has been constructed to satisfy this estimate and rules have been elaborated for its learning and testing. The learning process involves a small (of the order of 10%) part of the total set of promoter sequences. During this procedure the neural network develops a system of distinctive features (key words) to be used as a reference in identifying promoters against the background of random sequences. The learning quality is then tested with the whole set. The efficiency of promoter recognition has been found to amount to 94 to 99%. The probability of an arbitrary sequence being identified as a promoter is 2 to 6%.     

Localized states in an unbounded neural field equation with smooth firing rate function: a multi-parameter analysis

The existence of spatially localized solutions in neural networks is an important topic in neuroscience as these solutions are considered to characterize working (short-term) memory. We work with an unbounded neural network represented by the neural field equation with smooth firing rate function and a wizard hat spatial connectivity. Noting that stationary solutions of our neural field equation are equivalent to homoclinic orbits in a related fourth order ordinary differential equation, we apply normal form theory for a reversible Hopf bifurcation to prove the existence of localized solutions; further, we present results concerning their stability. Numerical continuation is used to compute branches of localized solution that exhibit snaking-type behaviour. We describe in terms of three parameters the exact regions for which localized solutions persist.     

A neural network methodology of quadratic optimization.

According to the basic optimization principle of artificial neural networks, a novel kind of neural network model for solving the quadratic programming problem is presented. The methodology is based on the Lagrange multiplier theory in optimization and seeks to provide solutions satisfying the necessary conditions of optimality. The equilibrium point of the network satisfies the Kuhn-Tucker condition for the problem. The stability and convergency of the neural network is investigated and the strategy of the neural optimization is discussed. The feasibility of the neural network method is verified with the computation examples. Results of the simulation of the neural network to solve optimum problems are presented to illustrate the computational power of the neural network method.     

Improving the performance of the Hopfield-Tank neural network through normalization and annealing

A simple formulation of the TSP energy function is described which, in combination with a normalized Hopfield-Tank neural network, eliminates the difficulty in finding valid tours. This technique is applicable to many other optimization problems involving n-way decisions (such as VLSI layout and resource allocation) and is easily implemented in a VLSI neural network. The solution quality is shown to be dependent on the formation of seed-points which are influenced by the constraint penalties and the temperature (i.e. the neural gain). Near-optimal tours are found by annealing the network down to a critical temperature at which a single seed-point is dominant. The seed-points and critical temperature (which also affect standard Hopfield network solutions to the TSP) can be predicted with reasonable accuracy. It is also shown that the annealing process is not necessary and good tours result if the network is allowed to converge solely at the critical temperature. The seed-points can be eliminated entirely by assigning different temperatures to groups of neurons such that the tour evolves uniformly throughout the cities. The resulting network finds the optimum tour in a 30-city example in 30% of the trials.     

Artificial neural networks and their use in quantitative pathology

A brief general introduction to artificial neural networks is presented, examining in detail the structure and operation of a prototype net developed for the solution of a simple pattern recognition problem in quantitative pathology. The process by which a neural network learns through example and gradually embodies its knowledge as a distributed representation is discussed, using this example. The application of neurocomputer technology to problems in quantitative pathology is explored, using real-world and illustrative examples. Included are examples of the use of artificial neural networks for pattern recognition, database analysis and machine vision. In the context of these examples, characteristics of neural nets, such as their ability to tolerate ambiguous, noisy and spurious data and spontaneously generalize from known examples to handle unfamiliar cases, are examined. Finally, the strengths and deficiencies of a connectionist approach are compared to those of traditional symbolic expert system methodology. It is concluded that artificial neural networks, used in conjunction with other nonalgorithmic artificial intelligence techniques and traditional algorithmic processing, may provide useful software engineering tools for the development of systems in quantitative pathology.     

Impact of network structure and cellular response on spike time correlations

Novel experimental techniques reveal the simultaneous activity of larger and larger numbers of neurons. As a result there is increasing interest in the structure of cooperative--or correlated--activity in neural populations, and in the possible impact of such correlations on the neural code. A fundamental theoretical challenge is to understand how the architecture of network connectivity along with the dynamical properties of single cells shape the magnitude and timescale of correlations. We provide a general approach to this problem by extending prior techniques based on linear response theory. We consider networks of general integrate-and-fire cells with arbitrary architecture, and provide explicit expressions for the approximate cross-correlation between constituent cells. These correlations depend strongly on the operating point (input mean and variance) of the neurons, even when connectivity is fixed. Moreover, the approximations admit an expansion in powers of the matrices that describe the network architecture. This expansion can be readily interpreted in terms of paths between different cells. We apply our results to large excitatory-inhibitory networks, and demonstrate first how precise balance--or lack thereof--between the strengths and timescales of excitatory and inhibitory synapses is reflected in the overall correlation structure of the network. We then derive explicit expressions for the average correlation structure in randomly connected networks. These expressions help to identify the important factors that shape coordinated neural activity in such networks.     

Neural model of the genetic network

Many cell control processes consist of networks of interacting elements that affect the state of each other over time. Such an arrangement resembles the principles of artificial neural networks, in which the state of a particular node depends on the combination of the states of other neurons. The lambda bacteriophage lysis/lysogeny decision circuit can be represented by such a network. It is used here as a model for testing the validity of a neural approach to the analysis of genetic networks. The model considers multigenic regulation including positive and negative feedback. It is used to simulate the dynamics of the lambda phage regulatory system; the results are compared with experimental observation. The comparison proves that the neural network model describes behavior of the system in full agreement with experiments; moreover, it predicts its function in experimentally inaccessible situations and explains the experimental observations. The application of the principles of neural networks to the cell control system leads to conclusions about the stability and redundancy of genetic networks and the cell functionality. Reverse engineering of the biochemical pathways from proteomics and DNA micro array data using the suggested neural network model is discussed.     

Decreasing-Rate Pruning Optimizes the Construction of Efficient and Robust Distributed Networks

Robust, efficient, and low-cost networks are advantageous in both biological and engineered systems. During neural network development in the brain, synapses are massively over-produced and then pruned-back over time. This strategy is not commonly used when designing engineered networks, since adding connections that will soon be removed is considered wasteful. Here, we show that for large distributed routing networks, network function is markedly enhanced by hyper-connectivity followed by aggressive pruning and that the global rate of pruning, a developmental parameter not previously studied by experimentalists, plays a critical role in optimizing network structure. We first used high-throughput image analysis techniques to quantify the rate of pruning in the mammalian neocortex across a broad developmental time window and found that the rate is decreasing over time. Based on these results, we analyzed a model of computational routing networks and show using both theoretical analysis and simulations that decreasing rates lead to more robust and efficient networks compared to other rates. We also present an application of this strategy to improve the distributed design of airline networks. Thus, inspiration from neural network formation suggests effective ways to design distributed networks across several domains.     

APPLICATION OF FUZZY SETS AND NEURAL NETWORKS IN SENSORY ANALYSIS

This paper describes an initial but fundamental attempt to lay some groundwork for a fuzzy-set-based paradigm for sensory analysis and to demonstrate how fuzzy set and neural network techniques may lead to a natural way for sensory data interpretation. Sensory scales are described as fuzzy sets, sensory attributes as fuzzy variables, and sensory responses as sample membership grades. Multi-judge responses are formulated as a fuzzy membership vector or fuzzy histogram of response, which gives an overall panel response free of the unverifiable assumptions implied in conventional approaches. Neural networks are used to provide an effective tool for modeling and analysis of sensory responses in their naturally fuzzy and complex forms. A maximum method of defuzzification is proposed to give a crisp grade of the majority opinion. Two applications in meat quality evaluation are used to demonstrate the use of the paradigm and procedure. It is hoped that this work will bring up some new ideas and generate interest in research on application of fuzzy sets and neural networks in sensory analysis.     

Cooperative recurrent modular neural networks for constrained optimization: a survey of models and applications

Constrained optimization problems arise in a wide variety of scientific and engineering applications. Since several single recurrent neural networks when applied to solve constrained optimization problems for real-time engineering applications have shown some limitations, cooperative recurrent neural network approaches have been developed to overcome drawbacks of these single recurrent neural networks. This paper surveys in details work on cooperative recurrent neural networks for solving constrained optimization problems and their engineering applications, and points out their standing models from viewpoint of both convergence to the optimal solution and model complexity. We provide examples and comparisons to shown advantages of these models in the given applications.     

A ternary logic model for recurrent neuromime networks with delay

 In contrast to popular recurrent artificial neural network (RANN) models, biological neural networks have unsymmetric structures and incorporate significant delays as a result of axonal propagation. Consequently, biologically inspired neural network models are more accurately described by nonlinear differential-delay equations rather than nonlinear ordinary differential equations (ODEs), and the standard techniques for studying the dynamics of RANNs are wholly inadequate for these models. This paper develops a ternary-logic based method for analyzing these networks. Key to the technique is the realization that a nonzero delay produces a bounded stability region. This result significantly simplifies the construction of sufficient conditions for characterizing the network equilibria. If the network gain is large enough, each equilibrium can be classified as either asymptotically stable or unstable. To illustrate the analysis technique, the swim central pattern generator (CPG) of the sea slug Tritonia diomedea is examined. For wide range of reasonable parameter values, the ternary analysis shows that none of the network equilibria are stable, and thus the network must oscillate. The results show that complex synaptic dynamics are not necessary for pattern generation. Received: 15 June 1994/Accepted in revised form: 10 February 1995