Successive learning in hetero-associative memory using chaotic neural networks

In this paper, we propose a successive learning method in hetero-associative memories, such as Bidirectional Associative Memories and Multidirectional Associative Memories, using chaotic neural networks. It can distinguish unknown data from the stored known data and can learn the unknown data successively. The proposed model makes use of the difference in the response to the input data in order to distinguish unknown data from the stored known data. When input data is regarded as unknown data, it is memorized. Furthermore, the proposed model can estimate and learn correct data from noisy unknown data or incomplete unknown data by considering the temporal summation of the continuous data input. In addition, similarity to the physiological facts in the olfactory bulb of a rabbit found by Freeman are observed in the behavior of the proposed model. A series of computer simulations shows the effectiveness of the proposed model.     

Improved backpropagation learning in neural networks with windowed momentum

Backpropagation, which is frequently used in Neural Network training, often takes a great deal of time to converge on an acceptable solution. Momentum is a standard technique that is used to speed up convergence and maintain generalization performance. In this paper we present the Windowed Momentum algorithm, which increases speedup over Standard Momentum. Windowed Momentum is designed to use a fixed width history of recent weight updates for each connection in a neural network. By using this additional information, Windowed Momentum gives significant speedup over a set of applications with same or improved accuracy. Windowed Momentum achieved an average speedup of 32% in convergence time on 15 data sets, including a large OCR data set with over 500,000 samples. In addition to this speedup, we present the consequences of sample presentation order. We show that Windowed Momentum is able to overcome these effects that can occur with poor presentation order and still maintain its speedup advantages.     

Adaptive networks using learning matrices

Summary The performance of the Learning Matrix (LM) is suitable for the design of adaptive networks of higher complexity. It has been published, how to connect a LM with a generator of patterns (binary or nonbinary) and a ring-counter to result in an automatic classification of the presented patterns. This paper describes, how to connect two LM's to form an Autonomous Learning Matrix Dipole (ALD) and how to organize it, so that it adapts itself to an environment according to a given evaluation scale. For this purpose, a third type of input (beside e and b), namely h seems to be useful. This h-input controls the rate of adaptation of the LM.Using such ALD's, one may design adaptive structures of even higher complexity, for example with an adaptive internal model.The principle of Learning Matrices has been explained in detail (see e.g. IEEE Transactions on Electronic Computers, Vol. EC-12, No. 6, December, 1963, pp. 846–862). Using such learning matrices (LM), one may build up adaptive networks with rather interesting functions. Perhaps they are interesting for the physiologist and psychologist as well as for the engineer. Let us first recall the most essential details of the LM's.

---

Zusammenfassung Die Funktion der Lernmatrix (LM) erlaubt den Entwurf adaptiver Netzwerke höherer Komplexität. Es wurde an anderer Stelle schon beschrieben, wie eine LM (binär oder nichtbinär) mit einem Generator für Eigenschaftssätze und einem Ringzähler zusammengeschaltet werden kann, um eine selbsttätige Klassifikation der angebotenen Eigenschaftssätze zu bewirken. Im vorliegenden Aufsatz wird erklärt, wie zwei LM so zusammengeschaltet werden können, dacß sich ein Autonomer Lernmatrix-Dipol (ALD) ergibt, und wie dieser zu organisieren ist, daß er sich einer gegebenen Außenwelt nach Maßgabe einer vorgegebenen Werteskala anpaßt. Zu diesem Zweck erweist sich außer den bisher beschriebenen beiden Zugangen zur LM (nämlich e und b) ein dritter sehr zweckmäßig, nämlich h. Dieser h-Eingang beeinflußt die Lerngeschwindigkeit der LM.Unter Verwendung solcher ALD's kann man adaptive Strukturen noch höherer Komplexität aufbauen, beispielsweise solche mit adaptivem innerem Modell.

---

---

Visiting Professor of Electrical Engineering Stanford University.     

Pruning artificial neural networks using neural complexity measures

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.     

Pollen texture identification using neural networks

The importance of research leading to the automation of pollen identification is briefly outlined. A new technique, neural network analysis, is briefly introduced, and then applied to the determination of light microscope images of pollen grains. The results are compared with some previously published statistical classifiers. Although both types of classifiers may work, the neural network is apparently superior to the statistical methods in three ways: high success rates (100% in this case), small number of samples needed for training, and simplicity of features.     

Modeling of pain using artificial neural networks

In dealing with human nervous system, the sensation of pain is as sophisticated as other physiological phenomena. To obtain an acceptable model of the pain, physiology of the pain has been analysed in the present paper. Pain mechanisms are explained in block diagram representation form. Because of the nonlinear interactions existing among different sections in the diagram, artificial neural networks (ANNs) have been exploited. The basic patterns associated with chronic and acute pain have been collected and then used to obtain proper features for training the neural networks. Both static and dynamic representations of the ANNs were used in this regard. The trained networks then were employed to predict response of the body when it is exposed to special excitations. These excitations have not been used in the training phase and their behavior is interesting from the physiological view. Some of these predictions can be inferred from clinical experimentations. However, more clinical tests have to be accomplished for some of the predictions.     

Evidence of hyperplanes in the genetic learning of neural networks

Genetic Algorithms have been successfully applied to the learning process of neural networks simulating artificial life. In previous research we compared mutation and crossover as genetic operators on neural networks directly encoded as real vectors (Manczer and Parisi 1990). With reference to crossover we were actually testing the building blocks hypothesis, as the effectiveness of recombination relies on the validity of such hypothesis. Even with the real genotype used, it was found that the average fitness of the population of neural networks is optimized much more quickly by crossover than it is by mutation. This indicated that the intrinsic parallelism of crossover is not reduced by the high cardinality, as seems reasonable and has indeed been suggested in GA theory (Antonisse 1989). In this paper we first summarize such findings and then propose an interpretation in terms of the spatial correlation of the fitness function with respect to the metric defined by the average steps of the genetic operators. Some numerical evidence of such interpretation is given, showing that the fitness surface appears smoother to crossover than it does to mutation. This confirms indirectly that crossover moves along privileged directions, and at the same time provides a geometric rationale for hyperplanes.     

Recognition of general patterns using neural networks

The Hopfield model of neural network stores memory in its symmetric synaptic connections and can only learn to recognize sets of nearly orthogonal patterns. A new algorithm is put forth to permit the recognition of general (non-orthogonal) patterns. The algorithm specifies the construction of the new network's memory matrix T _ij, which is, in general, asymmetrical and contains the Hopfield neural network (Hopfield 1982) as a special case. We find further that in addition to this new algorithm for general pattern recognition, there exists in fact a large class of T _ij memory matrices which permit the recognition of non-orthogonal patterns. The general form of this class of T _ij memory matrix is presented, and the projection matrix neural network (Personnaz et al. 1985) is found as a special case of this general form. This general form of memory matrix extends the library of memory matrices which allow a neural network to recognize non-orthogonal patterns. A neural network which followed this general form of memory matrix was modeled on a computer and successfully recognized a set of non-orthogonal patterns. The new network also showed a tolerance for altered and incomplete data. Through this new method, general patterns may be taught to the neural network.     

Synchrony based learning rule of Hopfield like chaotic neural networks with desirable structure

In this paper a new learning rule for the coupling weights tuning of Hopfield like chaotic neural networks is developed in such a way that all neurons behave in a synchronous manner, while the desirable structure of the network is preserved during the learning process. The proposed learning rule is based on sufficient synchronization criteria, on the eigenvalues of the weight matrix belonging to the neural network and the idea of Structured Inverse Eigenvalue Problem. Our developed learning rule not only synchronizes all neuron’s outputs with each other in a desirable topology, but also enables us to enhance the synchronizability of the networks by choosing the appropriate set of weight matrix eigenvalues. Specifically, this method is evaluated by performing simulations on the scale-free topology.     

Inferring species membership using DNA sequences with back-propagation neural networks

DNA barcoding as a method for species identification is rapidly increasing in popularity. However, there are still relatively few rigorous methodological tests of DNA barcoding. Current distance-based methods are frequently criticized for treating the nearest neighbor as the closest relative via a raw similarity score, lacking an objective set of criteria to delineate taxa, or for being incongruent with classical character-based taxonomy. Here, we propose an artificial intelligence-based approach - inferring species membership via DNA barcoding with back-propagation neural networks (named BP-based species identification) - as a new advance to the spectrum of available methods. We demonstrate the value of this approach with simulated data sets representing different levels of sequence variation under coalescent simulations with various evolutionary models, as well as with two empirical data sets of COI sequences from East Asian ground beetles (Carabidae) and Costa Rican skipper butterflies. With a 630-to 690-bp fragment of the COI gene, we identified 97.50% of 80 unknown sequences of ground beetles, 95.63%, 96.10%, and 100% of 275, 205, and 9 unknown sequences of the neotropical skipper butterfly to their correct species, respectively. Our simulation studies indicate that the success rates of species identification depend on the divergence of sequences, the length of sequences, and the number of reference sequences. Particularly in cases involving incomplete lineage sorting, this new BP-based method appears to be superior to commonly used methods for DNA-based species identification.     

Clustering with neural networks

Partitioning a set ofN patterns in ad-dimensional metric space intoK clusters — in a way that those in a given cluster are more similar to each other than the rest — is a problem of interest in many fields, such as, image analysis, taxonomy, astrophysics, etc. As there are approximatelyK ^N/K! possible ways of partitioning the patterns amongK clusters, finding the best solution is beyond exhaustive search whenN is large. We show that this problem, in spite of its exponential complexity, can be formulated as an optimization problem for which very good, but not necessarily optimal, solutions can be found by using a Hopfield model of neural networks. To obtain a very good solution, the network must start from many randomly selected initial states. The network is simulated on the MPP, a 128 × 128 SIMD array machine, where we use the massive parallelism not only in solving the differential equations that govern the evolution of the network, but also in starting the network from many initial states at once thus obtaining many solutions in one run. We achieve speedups of two to three orders of magnitude over serial implementations and the promise through Analog VLSI implementations of further speedups of three to six orders of magnitude.Supported by a National Research Council-NASA Research Associatship     

Reinforcement learning with internal expectation in the random neural networks for cascaded decisions

The reinforcement learning scheme proposed in Halici (J. Biosystems 40 (1997) 83) for the random neural network (RNN) (Neural Computation 1 (1989) 502) is based on reward and performs well for stationary environments. However, when the environment is not stationary it suffers from getting stuck to the previously learned action and extinction is not possible. To overcome the problem, the reinforcement scheme is extended in Halici (Eur. J. Oper. Res., 126(2000) 288) by introducing a new weight update rule (E-rule) which takes into consideration the internal expectation of reinforcement. Although the E-rule is proposed for the RNN, it can be used for training learning automata or other intelligent systems based on reinforcement learning. This paper looks into the behavior of the learning scheme with internal expectation for the environments where the reinforcement is obtained after a sequence of cascaded decisions. The simulation results have shown that the RNN learns well and extinction is possible even for the cases with several decision steps and with hundreds of possible decision paths.     

Bioprocess model building using artificial neural networks

Artificial neural networks are made upon of highly interconnected layers of simple neuron-like nodes. The neurons act as non-linear processing elements within the network. An attractive property of artificial neural networks is that given the appropriate network topology, they are capable of learning and characterising non-linear functional relationships. Furthermore, the structure of the resulting neural network based process model may be considered generic, in the sense that little prior process knowledge is required in its determination. The methodology therefore provides a cost efficient and reliable process modelling technique. One area where such a technique could be useful is biotechnological systems. Here, for example, the use of a process model within an estimation scheme has long been considered an effective means of overcoming inherent on-line measurement problems. However, the development of an accurate process model is extremely time consuming and often results in a model of limited applicability. Artificial neural networks could therefore prove to be a useful model building tool when striving to improve bioprocess operability. Two large scale industrial fermentation systems have been considered as test cases; a fed-batch penicillin fermentation and a continuous mycelial fermentation. Both systems serve to demonstrate the utility, flexibility and potential of the artificial neural network approach to process modelling.     

Content-based image retrieval system using neural networks

An effective image retrieval system is developed based on the use of neural networks (NNs). It takes advantages of association ability of multilayer NNs as matching engines which calculate similarities between a user's drawn sketch and the stored images. The NNs memorize pixel information of every size-reduced image (thumbnail) in the learning phase. In the retrieval phase, pixel information of a user's drawn rough sketch is inputted to the learned NNs and they estimate the candidates. Thus the system can retrieve candidates quickly and correctly by utilizing the parallelism and association ability of NNs. In addition, the system has learning capability: it can automatically extract features of a user's drawn sketch during the retrieval phase and can store them as additional information to improve the performance. The software for querying, including efficient graphical user interfaces, has been implemented and tested. The effectiveness of the proposed system has been investigated through various experimental tests.     

Route searching based on neural networks and heuristic reinforcement learning

In this paper, an improved and much stronger RNH-QL method based on RBF network and heuristic Q-learning was put forward for route searching in a larger state space. Firstly, it solves the problem of inefficiency of reinforcement learning if a given problem’s state space is increased and there is a lack of prior information on the environment. Secondly, RBF network as weight updating rule, reward shaping can give an additional feedback to the agent in some intermediate states, which will help to guide the agent towards the goal state in a more controlled fashion. Meanwhile, with the process of Q-learning, it is accessible to the underlying dynamic knowledge, instead of the need of background knowledge of an upper level RBF network. Thirdly, it improves the learning efficiency by incorporating the greedy exploitation strategy to train the neural network, which has been testified by the experimental results.     

A real time learning algorithm for recurrent analog neural networks

A new learning algorithm is described for a general class of recurrent analog neural networks which ultimately settle down to a steady state. Recently, Pineda (Pineda 1987; Almeida 1987; Ikeda et al. 1988) has introduced a learning rule for the recurrent net in which the connection weights are adjusted so that the distance between the stable outputs of the current system and the desired outputs will be maximally decreased. In this method, many cycles are needed in order to get a target system. In each cycle, the recurrent net is run until it reaches a stable state. After that, the weight change is calculated by using a linearized recurrent net which receives the current error of the system as a bias input. In the new algorithm the weights are changed so that the total error of neuron outputs over the entire trajectory is minimized. The weights are adjusted in real time when the network is running. In this method, the trajectory to the target system can be controlled, whereas Pineda's algorithm only controls the position of the fixed point. The relation to the back propagation method (Hinton et al. 1986) is also discussed.     

On-line prediction of fermentation variables using neural networks

This article presents an introduction to the use of neural network computational algorithms for the dynamic modeling of bioprocesses. The dynamic neural model is used for the prediction of key fermentation variables. This relatively hew method is compared with a more traditional prediction technique to judge its performance for prediction. Illustrative simulation results of a continuous stirred tank fermentor are used for this comparison. It is shown that neural network models are accurate with a certain degree of noise immunity. They offer the distinctive ability over more traditional methods to learn very naturally complex relationships without requiring the knowledge of the model structure.     

Prediction of beta-turns in proteins using neural networks

The use of neural networks to improve empirical secondary structure prediction is explored with regard to the identification of the position and conformational class of beta-turns, a four-residue chain reversal. Recently an algorithm was developed for beta-turn predictions based on the empirical approach of Chou and Fasman using different parameters for three classes (I, II and non-specific) of beta-turns. In this paper, using the same data, an alternative approach to derive an empirical prediction method is used based on neural networks which is a general learning algorithm extensively used in artificial intelligence. Thus the results of the two approaches can be compared. The most severe test of prediction accuracy is the percentage of turn predictions that are correct and the neural network gives an overall improvement from 20.6% to 26.0%. The proportion of correctly predicted residues is 71%, compared to a chance level of about 58%. Thus neural networks provide a method of obtaining more accurate predictions from empirical data than a simpler method of deriving propensities.     

Application of neural networks using the volume learning algorithm for the study of structure-activity relationship in chemical compounds

A volume learning algorithm for artificial neural networks was developed to quantitatively describe the three-dimensional structure-activity relationships using as an example N-benzylpiperidine derivatives. The new algorithm combines two types of neural networks, the Kohonen and the feed-forward artificial neural networks, which are used to analyze the input grid data generated by the comparative molecular field approach. Selection of the most informative parameters using the algorithm helped to reveal the most important spatial properties of the molecules, which affect their biological activities. Cluster regions determined using the new algorithm adequately predicted the activity of molecules from a control data set.     

Prediction of RNA secondary structure with pseudoknots using coupled deep neural networks

Noncoding RNAs play important roles in cell and their secondary structures are vital for understanding their tertiary structures and functions.Many prediction m...