Cross-talk theory of memory capacity in neural networks

The present paper presents a theory for the mechanics of cross-talk among constituent neurons in networks in which multiple memory traces have been embedded, and develops criteria for memory capacity based on the disruptive influences of this cross-talk. The theory is based on interconnection patterns defined by the sequential configuration model of dynamic firing patterns. The theory accurately predicts the memory capacities observed in computer simulated nets, and predicts that cortical-like modules should be able to store up to about 300–900 selectively retrievable memory traces before disruption by cross-talk is likely. It also predicts that the cortex may has designed itself for modules of 30,000 neurons to at least in part to optimize memory capacity.     

A new method for the re-implementation of threshold logic functions with cellular neural networks

A new strategy is presented for the implementation of threshold logic functions with binary-output Cellular Neural Networks (CNNs). The objective is to optimize the CNNs weights to develop a robust implementation. Hence, the concept of generative set is introduced as a convenient representation of any linearly separable Boolean function. Our analysis of threshold logic functions leads to a complete algorithm that automatically provides an optimized generative set. New weights are deduced and a more robust CNN template assuming the same function can thus be implemented. The strategy is illustrated by a detailed example.     

Codes, operations, measurements and neural networks

Numerous neural codes and primary neural operations (logical and arithmetical ones, mappings, transformations) were listed [e.g. Perkel, D., Bullock, T.H., 1968. Neurosci. Res. Program Bull 6, 221-348] during the past decades. None of them is ubiquitous or universal. In reality neural operations take place in continuous time and working with unreliable elements, but they still can be simulated with synchronized discrete time scales and chaotic models. Here, a possible neural mechanism, called 'measure like' code is introduced and examined. The neurons are regarded as measuring devices, dealing with 'measures', more or less in mathematical sense. The subadditivity--eminent property of measures--may be implemented with neuronal refractoriness and such synapses operate like particle counters with dead time. This hypothetical code is neither ubiquitous, nor universal, e.g. temporal summation (multiplication) causes just the opposite phenomenon, the supra-additivity also with respect to the number of spikes (anti-measures). This is a cause of more difficult neural implementation of OR gate, than that of the AND. Possibilities for transitional mechanisms (e.g. between traditional logical gates, etc.) are stressed here. Parameter tuning might change either code or operation.     

Using fuzzy logic to design fermentation media: A comparison to neural networks and factorial design

Summary In analysis of oil accumulation by the yeast Rhodotorula gracilis, fuzzy logic and neural networks were shown to be able to significantly reduce the number of experiments required in designing fermentation media. Fuzzy logic performed similarly, although slightly less accurately, than neural networks in predicting the outcome of shake flask experiments. In some instances having too many fuzzy rules decreased the accuracy of the technique, and further work is required to determine the criteria for achieving optimum performance from the fuzzy logic model. Overall fuzzy logic is a viable and useful tool for designing fermentation media.     

Determination of eukaryotic protein coding regions using neural networks and information theory.

Our previous work applied neural network techniques to the problem of discriminating open reading frame (ORF) sequences taken from introns versus exons. The method counted the codon frequencies in an ORF of a specified length, and then used this codon frequency representation of DNA fragments to train a neural net (essentially a Perceptron with a sigmoidal, or "soft step function", output) to perform this discrimination. After training, the network was then applied to a disjoint "predict" set of data to assess accuracy. The resulting accuracy in our previous work was 98.4%, exceeding accuracies reported in the literature at that time for other algorithms. Here, we report even higher accuracies stemming from calculations of mutual information (a correlation measure) of spatially separated codons in exons, and in introns. Significant mutual information exists in exons, but not in introns, between adjacent codons. This suggests that dicodon frequencies of adjacent codons are important for intron/exon discrimination. We report that accuracies obtained using a neural net trained on the frequency of dicodons is significantly higher at smaller fragment lengths than even our original results using codon frequencies, which were already higher than simple statistical methods that also used codon frequencies. We also report accuracies obtained from including codon and dicodon statistics in all six reading frames, i.e. the three frames on the original and complement strand. Inclusion of six-frame statistics increases the accuracy still further. We also compare these neural net results to a Bayesian statistical prediction method that assumes independent codon frequencies in each position. The performance of the Bayesian scheme is poorer than any of the neural based schemes, however many methods reported in the literature either explicitly, or implicitly, use this method. Specifically, Bayesian prediction schemes based on codon frequencies achieve 90.9% accuracy on 90 codon ORFs, while our best neural net scheme reaches 99.4% accuracy on 60 codon ORFs. "Accuracy" is defined as the average of the exon and intron sensitivities. Achievement of sufficiently high accuracies on short fragment lengths can be useful in providing a computational means of finding coding regions in unannotated DNA sequences such as those arising from the mega-base sequencing efforts of the Human Genome Project. We caution that the high accuracies reported here do not represent a complete solution to the problem of identifying exons in "raw" base sequences. The accuracies are considerably lower from exons of small length, although still higher than accuracies reported in the literature for other methods. Short exon lengths are not uncommon.(ABSTRACT TRUNCATED AT 400 WORDS)     

Artificial neural networks: fundamentals, computing, design, and application

Artificial neural networks (ANNs) are relatively new computational tools that have found extensive utilization in solving many complex real-world problems. The attractiveness of ANNs comes from their remarkable information processing characteristics pertinent mainly to nonlinearity, high parallelism, fault and noise tolerance, and learning and generalization capabilities. This paper aims to familiarize the reader with ANN-based computing (neurocomputing) and to serve as a useful companion practical guide and toolkit for the ANNs modeler along the course of ANN project development. The history of the evolution of neurocomputing and its relation to the field of neurobiology is briefly discussed. ANNs are compared to both expert systems and statistical regression and their advantages and limitations are outlined. A bird's eye review of the various types of ANNs and the related learning rules is presented, with special emphasis on backpropagation (BP) ANNs theory and design. A generalized methodology for developing successful ANNs projects from conceptualization, to design, to implementation, is described. The most common problems that BPANNs developers face during training are summarized in conjunction with possible causes and remedies. Finally, as a practical application, BPANNs were used to model the microbial growth curves of S. flexneri. The developed model was reasonably accurate in simulating both training and test time-dependent growth curves as affected by temperature and pH.     

Theoretical note: the C/T ratio in artificial neural networks

This paper describes computer simulations of the effect of the C/T ratio on acquisition rate in artificial neural networks. The networks consisted of neural processing elements that functioned according to a neurocomputational model whose learning rule is consistent with information on dopaminergic mechanisms of reinforcement. In Simulation 1, three comparisons were made: constant C and variable T, variable C and constant T, and a constant C/T with variable C and T. In the last two comparisons, C was manipulated by changing the probability of reinforcement within the intertrial interval (ITI), in the absence of the conditioned stimulus (CS). Acquisition rate tended to increase with C/T, and the invariant ratio had no effect. In Simulation 2, C was manipulated by changing the ITI, with continuous reinforcement in the presence of the CS and no reinforcements in its absence. Results were comparable to those obtained in Simulation 1. Simulation 3 further explored the effect of the invariant ratio, but with larger absolute values of C and T, which slowed acquisition significantly. The results parallel some experimental findings and theoretical implications of the Gibbon-Balsam model, showing that they can emerge from the moment-to-moment dynamics of a neural-network model. In contrast to that model, however, Simulation 3 suggests that the effect of invariant C/T ratios may be bounded.     

Delay, noise and phase locking in pulse coupled neural networks

This paper studies the effect of several delay times and noise on the stability of the phase-locked state in the lighthouse model and the integrate and fire model of a pulse coupled neural network. The coupling between neurons may be arbitrary. In both models the increase of delay times leads to a weakening of the stability and to the occurrence of relaxation oscillations.     

Image analysis, neural networks, and the taxonomic impediment to biodiversity studies

The taxonomic impediment to biodiversity studies may be influenced radically by the application of new technology, in particular, desktop image analysers and neural networks. The former offer an opportunity to automate objective feature measurement processes, and the latter provide powerful pattern recognition and data analysis tools which are able to 'learn' patterns in multivariate data. The coupling of these technologies may provide a realistic opportunity for the automation of routine species identifications. The potential benefits and limitations of these technologies, along with the development of automated identification systems are reviewed.     

Sequential RAM-based neural networks: learnability, generalisation, knowledge extraction, and grammatical inference

A fundamental question in the field of artificial neural networks is what set of problems a given class of networks can perform (computability). Such a problem can be made less general, but no less important, by asking what these networks could learn by using a given training procedure (learnability). The basic purpose of this paper is to address the learnability problem. Specifically, it analyses the learnability of sequential RAM-based neural networks. The analytical tools used are those of Automata Theory. In this context, this paper establishes which class of problems and under what conditions such networks, together with their existing learning rules, can learn and generalize. This analysis also yields techniques for both extracting knowledge from and inserting knowledge into the networks. The results presented here, besides helping in a better understanding of the temporal behaviour of sequential RAM-based networks, could also provide useful insights for the integration of the symbolic/connectionist paradigms.     

Regulation of rhythm genesis by volume-limited,astroglia-like signals in neural networks

Rhythmic activity of the brain often depends on synchronized spiking of interneuronal networks interacting with principal neurons. The quest for physiological mechanisms regulating network synchronization has therefore been firmly focused on synaptic circuits. However, it has recently emerged that synaptic efficacy could be influenced by astrocytes that release signalling molecules into their macroscopic vicinity. To understand how this volume-limited synaptic regulation can affect oscillations in neural populations, here we explore an established artificial neural network mimicking hippocampal basket cells receiving inputs from pyramidal cells. We find that network oscillation frequencies and average cell firing rates are resilient to changes in excitatory input even when such changes occur in a significant proportion of participating interneurons, be they randomly distributed or clustered in space. The astroglia-like, volume-limited regulation of excitatory synaptic input appears to better preserve network synchronization (compared with a similar action evenly spread across the network) while leading to a structural segmentation of the network into cell subgroups with distinct firing patterns. These observations provide us with some previously unknown insights into the basic principles of neural network control by astroglia.     

Integrate and fire neural networks, piecewise contractive maps and limit cycles

We study the global dynamics of integrate and fire neural networks composed of an arbitrary number of identical neurons interacting by inhibition and excitation. We prove that if the interactions are strong enough, then the support of the stable asymptotic dynamics consists of limit cycles. We also find sufficient conditions for the synchronization of networks containing excitatory neurons. The proofs are based on the analysis of the equivalent dynamics of a piecewise continuous Poincaré map associated to the system. We show that for efficient interactions the Poincaré map is piecewise contractive. Using this contraction property, we prove that there exist a countable number of limit cycles attracting all the orbits dropping into the stable subset of the phase space. This result applies not only to the Poincaré map under study, but also to a wide class of general n-dimensional piecewise contractive maps.     

Feature decomposition architectures for neural networks: algorithms, error bounds, and applications

In recent years, systems consisting of multiple modular neural networks have attracted substantial interest in the neural networks community because of various advantages they offer over a single large monolithic network. In this paper, we propose two basic feature decomposition models (namely, parallel model and tandem model) in which each of the neural network modules processes a disjoint subset of the input features. A novel feature decomposition algorithm is introduced to partition the input space into disjoint subsets solely based on the available training data. Under certain assumptions, the approximation error due to decomposition can be proved to be bounded by any desired small value over a compact set. Finally, the performance of feature decomposition networks is compared with that of a monolithic network in real world bench mark pattern recognition and modeling problems.     

An FPGA hardware/software co-design towards evolvable spiking neural networks for robotics application

This paper presents an approach that permits the effective hardware realization of a novel Evolvable Spiking Neural Network (ESNN) paradigm on Field Programmable Gate Arrays (FPGAs). The ESNN possesses a hybrid learning algorithm that consists of a Spike Timing Dependent Plasticity (STDP) mechanism fused with a Genetic Algorithm (GA). The design and implementation direction utilizes the latest advancements in FPGA technology to provide a partitioned hardware/software co-design solution. The approach achieves the maximum FPGA flexibility obtainable for the ESNN paradigm. The algorithm was applied as an embedded intelligent system robotic controller to solve an autonomous navigation and obstacle avoidance problem.     

Artificial neural networks in models of specialization, guild evolution and sympatric speciation

Sympatric speciation can arise as a result of disruptive selection with assortative mating as a pleiotropic by-product. Studies on host choice, employing artificial neural networks as models for the host recognition system in exploiters, illustrate how disruptive selection on host choice coupled with assortative mating can arise as a consequence of selection for specialization. Our studies demonstrate that a generalist exploiter population can evolve into a guild of specialists with an 'ideal free' frequency distribution across hosts. The ideal free distribution arises from variability in host suitability and density-dependent exploiter fitness on different host species. Specialists are less subject to inter-phenotypic competition than generalists and to harmful mutations that are common in generalists exploiting multiple hosts.When host signals used as cues by exploiters coevolve with exploiter recognition systems, our studies show that evolutionary changes may be continuous and cyclic. Selection changes back and forth between specialization and generalization in the exploiters, and weak and strong mimicry in the hosts, where non-defended hosts use the host investing in defence as a model. Thus, host signals and exploiter responses are engaged in a red-queen mimicry process that is ultimately cyclic rather then directional. In one phase, evolving signals of exploitable hosts mimic those of hosts less suitable for exploitation (i.e. the model). Signals in the model hosts also evolve through selection to escape the mimic and its exploiters. Response saturation constraints in the model hosts lead to the mimic hosts finally perfecting its mimicry, after which specialization in the exploiter guild is lost. This loss of exploiter specialization provides an opportunity for the model hosts to escape their mimics. Therefore, this cycle then repeats.We suggest that a species can readily evolve sympatrically when disruptive selection for specialization on hosts is the first step. In a sexual reproduction setting, partial reproductive isolation may first evolve by mate choice being confined to individuals on the same host. Secondly, this disruptive selection will favour assortative mate choice on genotype, thereby leading to increased reproductive isolation.     

Mean-field equations,bifurcation map and chaos in discrete time,continuous state,random neural networks

The dynamical behaviour of a very general model of neural networks with random asymmetric synaptic weights is investigated in the presence of random thresholds. Using mean-field equations, the bifurcations of the fixed points and the change of regime when varying control parameters are established. Different areas with various regimes are defined in the parameter space. Chaos arises generically by a quasi-periodicity route.     

Visual illusions in neural networks: Line neutralization,tilt after effect,and angle expansion

Certain visual illusions occur in neural networks that are capable of storing partially contrasted enhanced spatial patterns in short term memory (STM), and whose feature detectors are interconnected by nontrivial generalization gradients. These include neutralization, or adaptation, of nearly vertical or horizontal lines, tilt after-effect of successively viewed lines, and perceived angle expansion. Neutralization can be achieved by networks whose vertical and horizontal representations have higher saturation levels, broader tuning curves, or stronger input pathways. Tilt after-effect and angle expansion can be achieved by shunting lateral inhibition that causes an outward peak shift in the orientationally-coded STM pattern. The amount of outward peak shift is also dependent on the size of the potassium equilibrium point. Differences between the directions of tilt aftereffect (successive contrast) and angle expansion (simultaneous contrast) are ascribed to a normalization of total activity in the STM buffer whereby present stimuli and representations in STM of past stimuli interact to form a consistent action-oriented consensus.