Pruning artificial neural networks using neural complexity measures

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.     

Dynamical complexity in cognitive neural networks

In the last twenty years an important effort in brain sciences, especially in cognitive science, has been the development of mathematical tool that can deal with the complexity of extensive recordings corresponding to the neuronal activity obtained from hundreds of neurons. We discuss here along with some historical issues, advantages and limitations of Artificial Neural Networks (ANN) that can help to understand how simple brain circuits work and whether ANN can be helpful to understand brain neural complexity.     

The complexity of linkage analysis with neural networks

As the focus of genome-wide scans for disease loci have shifted from simple Mendelian traits to genetically complex traits, researchers have begun to consider new alternative ways to detect linkage that will consider more than the marginal effects of a single disease locus at a time. One interesting new method is to train a neural network on a genome-wide data set in order to search for the best non-linear relationship between identity-by-descent sharing among affected siblings at markers and their disease status. We investigate here the repeatability of the neural network results from run to run, and show that the results obtained by multiple runs of the neural network method may differ quite a bit. This is most likely due to the fact that training a neural network involves minimizing an error function with a multitude of local minima.     

The evolution of social learning rules: Payoff-biased and frequency-dependent biased transmission

Humans and other animals do not use social learning indiscriminately, rather, natural selection has favoured the evolution of social learning rules that make selective use of social learning to acquire relevant information in a changing environment. We present a gene-culture coevolutionary analysis of a small selection of such rules (unbiased social learning, payoff-biased social learning and frequency-dependent biased social learning, including conformism and anti-conformism) in a population of asocial learners where the environment is subject to a constant probability of change to a novel state. We define conditions under which each rule evolves to a genetically polymorphic equilibrium. We find that payoff-biased social learning may evolve under high levels of environmental variation if the fitness benefit associated with the acquired behaviour is either high or low but not of intermediate value. In contrast, both conformist and anti-conformist biases can become fixed when environment variation is low, whereupon the mean fitness in the population is higher than for a population of asocial learners. Our examination of the population dynamics reveals stable limit cycles under conformist and anti-conformist biases and some highly complex dynamics including chaos. Anti-conformists can out-compete conformists when conditions favour a low equilibrium frequency of the learned behaviour. We conclude that evolution, punctuated by the repeated successful invasion of different social learning rules, should continuously favour a reduction in the equilibrium frequency of asocial learning, and propose that, among competing social learning rules, the dominant rule will be the one that can persist with the lowest frequency of asocial learning.     

Artificial neural networks and the study of evolution of prey coloration

In this paper, I investigate the use of artificial neural networks in the study of prey coloration. I briefly review the anti-predator functions of prey coloration and describe both in general terms and with help of two studies as specific examples the use of neural network models in the research on prey coloration. The first example investigates the effect of visual complexity of background on evolution of camouflage. The second example deals with the evolutionary choice of defence strategy, crypsis or aposematism. I conclude that visual information processing by predators is central in evolution of prey coloration. Therefore, the capability to process patterns as well as to imitate aspects of predator's information processing and responses to visual information makes neural networks a well-suited modelling approach for the study of prey coloration. In addition, their suitability for evolutionary simulations is an advantage when complex or dynamic interactions are modelled. Since not all behaviours of neural network models are necessarily biologically relevant, it is important to validate a neural network model with empirical data. Bringing together knowledge about neural networks with knowledge about topics of prey coloration would provide a potential way to deepen our understanding of the specific appearances of prey coloration.     

Evidence of hyperplanes in the genetic learning of neural networks

Genetic Algorithms have been successfully applied to the learning process of neural networks simulating artificial life. In previous research we compared mutation and crossover as genetic operators on neural networks directly encoded as real vectors (Manczer and Parisi 1990). With reference to crossover we were actually testing the building blocks hypothesis, as the effectiveness of recombination relies on the validity of such hypothesis. Even with the real genotype used, it was found that the average fitness of the population of neural networks is optimized much more quickly by crossover than it is by mutation. This indicated that the intrinsic parallelism of crossover is not reduced by the high cardinality, as seems reasonable and has indeed been suggested in GA theory (Antonisse 1989). In this paper we first summarize such findings and then propose an interpretation in terms of the spatial correlation of the fitness function with respect to the metric defined by the average steps of the genetic operators. Some numerical evidence of such interpretation is given, showing that the fitness surface appears smoother to crossover than it does to mutation. This confirms indirectly that crossover moves along privileged directions, and at the same time provides a geometric rationale for hyperplanes.     

BioJazz: in silico evolution of cellular networks with unbounded complexity using rule-based modeling

Systems biologists aim to decipher the structure and dynamics of signaling and regulatory networks underpinning cellular responses; synthetic biologists can use this insight to alter existing networks or engineer de novo ones. Both tasks will benefit from an understanding of which structural and dynamic features of networks can emerge from evolutionary processes, through which intermediary steps these arise, and whether they embody general design principles. As natural evolution at the level of network dynamics is difficult to study, in silico evolution of network models can provide important insights. However, current tools used for in silico evolution of network dynamics are limited to ad hoc computer simulations and models. Here we introduce BioJazz, an extendable, user-friendly tool for simulating the evolution of dynamic biochemical networks. Unlike previous tools for in silico evolution, BioJazz allows for the evolution of cellular networks with unbounded complexity by combining rule-based modeling with an encoding of networks that is akin to a genome. We show that BioJazz can be used to implement biologically realistic selective pressures and allows exploration of the space of network architectures and dynamics that implement prescribed physiological functions. BioJazz is provided as an open-source tool to facilitate its further development and use. Source code and user manuals are available at: http://oss-lab.github.io/biojazz and http://osslab.lifesci.warwick.ac.uk/BioJazz.aspx.     

Concept formation in neural networks: Implications for evolution of cognitive functions

Parallel systems composed of many interconnected elements are both simple brain models and possible novel computer architectures. Potential advantages of such systems are massive parallelism with resulting speedup of computation as well as general ability to compute with noisy, corrupted, or missing data. Parallel, distributed, associative models have pronounced psychologies. Some ways of handling information are natural for them, and some things that we might want them to do are unnatural and quite difficult to do. A question of considerable interest is whether the models’ capabilities and limitations are features of human psychology. Such systems form categories based on the structure their inputs and display behavior that looks as if they form and use simple concepts. However, if noisy examples are learned, an initially stable concept structure may break up. One very simple function of names attached to categories — i.e. a rudimentary language — could be to stabilize a concept structure against fragmentation. In addition, if the statistical structure of the names reflects the statistical structure of the inputs, capacity and reliability of categorization and recognition is enhanced.     

Route searching based on neural networks and heuristic reinforcement learning

In this paper, an improved and much stronger RNH-QL method based on RBF network and heuristic Q-learning was put forward for route searching in a larger state space. Firstly, it solves the problem of inefficiency of reinforcement learning if a given problem’s state space is increased and there is a lack of prior information on the environment. Secondly, RBF network as weight updating rule, reward shaping can give an additional feedback to the agent in some intermediate states, which will help to guide the agent towards the goal state in a more controlled fashion. Meanwhile, with the process of Q-learning, it is accessible to the underlying dynamic knowledge, instead of the need of background knowledge of an upper level RBF network. Thirdly, it improves the learning efficiency by incorporating the greedy exploitation strategy to train the neural network, which has been testified by the experimental results.     

The meaning of the Weber-Fechner law and description of scenes in terms of neural networks

An attempt was made to determine to which functions of the visual cortex the Weber-Fechner law pertains. Pairs of lines were presented in different halves of the visual field. One of them was a reference line with an unchanged length, and the other (the test line) was of variable length. Psychometric curves reflecting the probability of the answer that the test line was longer than the reference one were plotted. A neuronal scheme for calculating the subjective reference stimulus, which differed from the objective reference stimulus, was proposed. The central zone of the visual field was determined. Two psychometric curves were obtained in the zone of each hemisphere. One of them was based on the results of tests where the test stimulus was presented above the reference stimulus and the other, on those where the test stimulus was below the reference one. The mutual positions of the curves were asymmetric in the hemispheres. One psychometric curve was obtained outside the central zone in each hemisphere. There was no dependence on the positions of the test and reference stimuli. The obtained data, according to which ΔL = const in the central zone and ΔL/L = const outside it, served as the basis for postulating the existence of three neuronal mechanisms. One of them is responsible for the interaction of neural networks between the hemispheres and serves for describing the scene, estimating the perspective, and determining the relative distances between objects. The second and third mechanisms are responsible for the interactions within the left and right hemispheres and serve only for describing the scene. Only the mechanism of describing the scene operates outside the central zone. It is assumed that the three postulated mechanisms, together with the mechanism of image recognition, create the visual image of the world perceived by the brain.     

Improved backpropagation learning in neural networks with windowed momentum

Backpropagation, which is frequently used in Neural Network training, often takes a great deal of time to converge on an acceptable solution. Momentum is a standard technique that is used to speed up convergence and maintain generalization performance. In this paper we present the Windowed Momentum algorithm, which increases speedup over Standard Momentum. Windowed Momentum is designed to use a fixed width history of recent weight updates for each connection in a neural network. By using this additional information, Windowed Momentum gives significant speedup over a set of applications with same or improved accuracy. Windowed Momentum achieved an average speedup of 32% in convergence time on 15 data sets, including a large OCR data set with over 500,000 samples. In addition to this speedup, we present the consequences of sample presentation order. We show that Windowed Momentum is able to overcome these effects that can occur with poor presentation order and still maintain its speedup advantages.     

Successive learning in hetero-associative memory using chaotic neural networks

In this paper, we propose a successive learning method in hetero-associative memories, such as Bidirectional Associative Memories and Multidirectional Associative Memories, using chaotic neural networks. It can distinguish unknown data from the stored known data and can learn the unknown data successively. The proposed model makes use of the difference in the response to the input data in order to distinguish unknown data from the stored known data. When input data is regarded as unknown data, it is memorized. Furthermore, the proposed model can estimate and learn correct data from noisy unknown data or incomplete unknown data by considering the temporal summation of the continuous data input. In addition, similarity to the physiological facts in the olfactory bulb of a rabbit found by Freeman are observed in the behavior of the proposed model. A series of computer simulations shows the effectiveness of the proposed model.     

Biologically plausible learning in neural networks: a lesson from bacterial chemotaxis

Learning processes in the brain are usually associated with plastic changes made to optimize the strength of connections between neurons. Although many details related to biophysical mechanisms of synaptic plasticity have been discovered, it is unclear how the concurrent performance of adaptive modifications in a huge number of spatial locations is organized to minimize a given objective function. Since direct experimental observation of even a relatively small subset of such changes is not feasible, computational modeling is an indispensable investigation tool for solving this problem. However, the conventional method of error back-propagation (EBP) employed for optimizing synaptic weights in artificial neural networks is not biologically plausible. This study based on computational experiments demonstrated that such optimization can be performed rather efficiently using the same general method that bacteria employ for moving closer to an attractant or away from a repellent. With regard to neural network optimization, this method consists of regulating the probability of an abrupt change in the direction of synaptic weight modification according to the temporal gradient of the objective function. Neural networks utilizing this method (regulation of modification probability, RMP) can be viewed as analogous to swimming in the multidimensional space of their parameters in the flow of biochemical agents carrying information about the optimality criterion. The efficiency of RMP is comparable to that of EBP, while RMP has several important advantages. Since the biological plausibility of RMP is beyond a reasonable doubt, the RMP concept provides a constructive framework for the experimental analysis of learning in natural neural networks.     

Embedding a grammatical description in deterministic chaos: an experiment in recurrent neural learning

We consider the modeling process of a biological agent by combining the concepts of neuroinformatics and deterministic chaos. We assume that an agent observes a target process as a stochastic symbolic process, which is restricted by grammatical constraints. Our main hypothesis is that an agent would learn the target model by reconstructing an equivalent quasi-stochastic process on its deterministic neural dynamics. We employed a recurrent neural network (RNN), which is regarded as an adjustable deterministic dynamical system. Then, we conducted an experiment to observe how the RNN learns to reconstruct the target process, represented by a stochastic finite state machine in the simulation. The result revealed the capability of the RNN to evolve, by means of learning, toward chaos, which is able to mimic a target's stochastic process. We precisely analyzed the evolutionary process as well as the internal representation of the neural dynamics obtained. This analysis enabled us to clarify an interesting mechanism of the self-organization of chaos by means of neural learning, and also showed how grammar can be embedded in the evolved deterministic chaos.     

Cognition and evolution: learning and the evolution of sex traits

The evolution of gender characteristics is an outcome of mate choice, which has been assumed to be genetically mediated. Recent research suggests that learning also has a role to play as an agent of sexual selection.