Training simultaneous recurrent neural network with resilient propagation for static optimization

This paper proposes a non-recurrent training algorithm, resilient propagation, for the Simultaneous Recurrent Neural network operating in relaxation-mode for computing high quality solutions of static optimization problems. Implementation details related to adaptation of the recurrent neural network weights through the non-recurrent training algorithm, resilient backpropagation, are formulated through an algebraic approach. Performance of the proposed neuro-optimizer on a well-known static combinatorial optimization problem, the Traveling Salesman Problem, is evaluated on the basis of computational complexity measures and, subsequently, compared to performance of the Simultaneous Recurrent Neural network trained with the standard backpropagation, and recurrent backpropagation for the same static optimization problem. Simulation results indicate that the Simultaneous Recurrent Neural network trained with the resilient backpropagation algorithm is able to locate superior quality solutions through comparable amount of computational effort for the Traveling Salesman Problem.     

Template learning of cellular neural network using genetic programming

A new learning algorithm for space invariant Uncoupled Cellular Neural Network is introduced. Learning is formulated as an optimization problem. Genetic Programming has been selected for creating new knowledge because they allow the system to find new rules both near to good ones and far from them, looking for unknown good control actions. According to the lattice Cellular Neural Network architecture, Genetic Programming will be used in deriving the Cloning Template. Exploration of any stable domain is possible by the current approach. Details of the algorithm are discussed and several application results are shown.     

The simultaneous recurrent neural network for addressing the scaling problem in static optimization.

A trainable recurrent neural network, Simultaneous Recurrent Neural network, is proposed to address the scaling problem faced by neural network algorithms in static optimization. The proposed algorithm derives its computational power to address the scaling problem through its ability to "learn" compared to existing recurrent neural algorithms, which are not trainable. Recurrent backpropagation algorithm is employed to train the recurrent, relaxation-based neural network in order to associate fixed points of the network dynamics with locally optimal solutions of the static optimization problems. Performance of the algorithm is tested on the NP-hard Traveling Salesman Problem in the range of 100 to 600 cities. Simulation results indicate that the proposed algorithm is able to consistently locate high-quality solutions for all problem sizes tested. In other words, the proposed algorithm scales demonstrably well with the problem size with respect to quality of solutions and at the expense of increased computational cost for large problem sizes.     

Artificial metaplasticity neural network applied to credit scoring

The assessment of the risk of default on credit is important for financial institutions. Different Artificial Neural Networks (ANN) have been suggested to tackle the credit scoring problem, however, the obtained error rates are often high. In the search for the best ANN algorithm for credit scoring, this paper contributes with the application of an ANN Training Algorithm inspired by the neurons' biological property of metaplasticity. This algorithm is especially efficient when few patterns of a class are available, or when information inherent to low probability events is crucial for a successful application, as weight updating is overemphasized in the less frequent activations than in the more frequent ones. Two well-known and readily available such as: Australia and German data sets has been used to test the algorithm. The results obtained by AMMLP shown have been superior to state-of-the-art classification algorithms in credit scoring.     

Artificial neural network based soft estimator for estimation of transducer static nonlinearity

In this work, the development of an Artificial Neural Network (ANN) based soft estimator is reported for the estimation of static-nonlinearity associated with the transducers. Under the realm of ANN based transducer modeling, only two neural models have been suggested to estimate the static-nonlinearity associated with the transducers with quite successful results. The first existing model is based on the concept of a functional link artificial neural network (FLANN) trained with mu-LMS (Least Mean Squares) learning algorithm. The second one is based on the architecture of a single layer linear ANN trained with alpha-LMS learning algorithm. However, both these models suffer from the problem of slow convergence (learning). In order to circumvent this problem, it is proposed to synthesize the direct model of transducers using the concept of a Polynomial-ANN (polynomial artificial neural network) trained with Levenberg-Marquardt (LM) learning algorithm. The proposed Polynomial-ANN oriented transducer model is implemented based on the topology of a single-layer feed-forward back-propagation-ANN. The proposed neural modeling technique provided an extremely fast convergence speed with increased accuracy for the estimation of transducer static nonlinearity. The results of convergence are very stimulating with the LM learning algorithm.     

An FPGA hardware/software co-design towards evolvable spiking neural networks for robotics application

This paper presents an approach that permits the effective hardware realization of a novel Evolvable Spiking Neural Network (ESNN) paradigm on Field Programmable Gate Arrays (FPGAs). The ESNN possesses a hybrid learning algorithm that consists of a Spike Timing Dependent Plasticity (STDP) mechanism fused with a Genetic Algorithm (GA). The design and implementation direction utilizes the latest advancements in FPGA technology to provide a partitioned hardware/software co-design solution. The approach achieves the maximum FPGA flexibility obtainable for the ESNN paradigm. The algorithm was applied as an embedded intelligent system robotic controller to solve an autonomous navigation and obstacle avoidance problem.     

Unsupervised fuzzy clustering using Weighted Incremental Neural Networks

A new more efficient variant of a recently developed algorithm for unsupervised fuzzy clustering is introduced. A Weighted Incremental Neural Network (WINN) is introduced and used for this purpose. The new approach is called FC-WINN (Fuzzy Clustering using WINN). The WINN algorithm produces a net of nodes connected by edges, which reflects and preserves the topology of the input data set. Additional weights, which are proportional to the local densities in input space, are associated with the resulting nodes and edges to store useful information about the topological relations in the given input data set. A fuzziness factor, proportional to the connectedness of the net, is introduced in the system. A watershed-like procedure is used to cluster the resulting net. The number of the resulting clusters is determined by this procedure. Only two parameters must be chosen by the user for the FC-WINN algorithm to determine the resolution and the connectedness of the net. Other parameters that must be specified are those which are necessary for the used incremental neural network, which is a modified version of the Growing Neural Gas algorithm (GNG). The FC-WINN algorithm is computationally efficient when compared to other approaches for clustering large high-dimensional data sets.     

Constructive Autoassociative Neural Network for Facial Recognition

Autoassociative artificial neural networks have been used in many different computer vision applications. However, it is difficult to define the most suitable neural network architecture because this definition is based on previous knowledge and depends on the problem domain. To address this problem, we propose a constructive autoassociative neural network called CANet (Constructive Autoassociative Neural Network). CANet integrates the concepts of receptive fields and autoassociative memory in a dynamic architecture that changes the configuration of the receptive fields by adding new neurons in the hidden layer, while a pruning algorithm removes neurons from the output layer. Neurons in the CANet output layer present lateral inhibitory connections that improve the recognition rate. Experiments in face recognition and facial expression recognition show that the CANet outperforms other methods presented in the literature.     

FPGA implementation of a pyramidal Weightless Neural Networks learning system

A hardware architecture of a Probabilistic Logic Neuron (PLN) is presented. The suggested model facilitates the on-chip learning of pyramidal Weightless Neural Networks using a modified probabilistic search reward/penalty training algorithm. The penalization strategy of the training algorithm depends on a predefined parameter called the probabilistic search interval. A complete Weightless Neural Network (WNN) learning system is modeled and implemented on Xilinx XC4005E Field Programmable Gate Array (FPGA), allowing its architecture to be configurable. Various experiments have been conducted to examine the feasibility and performance of the WNN learning system. Results show that the system has a fast convergence rate and good generalization ability.     

Predicting protein solubility with a hybrid approach by pseudo amino acid composition

Protein solubility plays a major role for understanding the crystal growth and crystallization process of protein. How to predict the propensity of a protein to be soluble or to form inclusion body is a long but not fairly resolved problem. After choosing almost 10,000 protein sequences from NCBI database and eliminating the sequences with 90% homologous similarity by CD-HIT, 5692 sequences remained. By using Chou's pseudo amino acid composition features, we predict the soluble protein with the three methods: support vector machine (SVM), back propagation neural network (BP Neural Network) and hybrid method based on SVM and BP Neural Network, respectively. Each method is evaluated by re-substitution test and 10-fold cross-validation test. In the re-substitution test, the BP Neural Network performs with the best results, in which the accuracy achieves 0.9288 and Matthews Correlation Coefficient (MCC) achieves 0.8513. Meanwhile, the other two methods are better than BP Neural Network in 10-fold cross-validation test. The hybrid method based on SVM and BP Neural Network is the best. The average accuracy is 0.8678 and average MCC is 0.7233. Although all of the three methods achieve considerable evaluations, the hybrid method is deemed to be the best, according to the performance comparison.     

Span: spike pattern association neuron for learning spatio-temporal spike patterns

Spiking Neural Networks (SNN) were shown to be suitable tools for the processing of spatio-temporal information. However, due to their inherent complexity, the formulation of efficient supervised learning algorithms for SNN is difficult and remains an important problem in the research area. This article presents SPAN - a spiking neuron that is able to learn associations of arbitrary spike trains in a supervised fashion allowing the processing of spatio-temporal information encoded in the precise timing of spikes. The idea of the proposed algorithm is to transform spike trains during the learning phase into analog signals so that common mathematical operations can be performed on them. Using this conversion, it is possible to apply the well-known Widrow-Hoff rule directly to the transformed spike trains in order to adjust the synaptic weights and to achieve a desired input/output spike behavior of the neuron. In the presented experimental analysis, the proposed learning algorithm is evaluated regarding its learning capabilities, its memory capacity, its robustness to noisy stimuli and its classification performance. Differences and similarities of SPAN regarding two related algorithms, ReSuMe and Chronotron, are discussed.     

Higherorder neural network group models for financial simulation

Real world financial data is often discontinuous and non-smooth. If we attempt to use neural networks to simulate such functions, then accuracy will be a problem. Neural network group models perform this function much better. Both Polynomial Higher Order Neural network Group (PHONG) and Trigonometric polynomial Higher Order Neural network Group (THONG) models are developed. These HONG models are open box, convergent models capable of approximating any kind of piecewise continuous function, to any degree of accuracy. Moreover they are capable of handling higher frequency, higher order non-linear and discontinuous data. Results obtained using a Higher Order Neural network Group financial simulator are presented, which confirm that HONG group models converge without difficulty, and are considerably more accurate than neural network models (more specifically, around twice as good for prediction, and a factor of four improvement in the case of simulation).     

Using weighted fixed neural networks for unsupervised fuzzy clustering

A novel algorithm for unsupervised fuzzy clustering is introduced. The algorithm uses a so-called Weighted Fixed Neural Network (WFNN) to store important and useful information about the topological relations in a given data set. The algorithm produces a weighted connected net, of weighted nodes connected by weighted edges, which reflects and preserves the topology of the input data set. The weights of the nodes and the edges in the resulting net are proportional to the local densities of data samples in input space. The connectedness of the net can be changed, and the higher the connectedness of the net is chosen, the fuzzier the system becomes. The new algorithm is computationally efficient when compared to other existing methods for clustering multi-dimensional data, such as color images.     

An STDP training algorithm for a spiking neural network with dynamic threshold neurons

This paper proposes a supervised training algorithm for Spiking Neural Networks (SNNs) which modifies the Spike Timing Dependent Plasticity (STDP)learning rule to support both local and network level training with multiple synaptic connections and axonal delays. The training algorithm applies the rule to two and three layer SNNs, and is benchmarked using the Iris and Wisconsin Breast Cancer (WBC) data sets. The effectiveness of hidden layer dynamic threshold neurons is also investigated and results are presented.     

应用神经网络和多谱段遥感数据对大屿山岛土地覆盖分类的研究

 土地覆盖是植物群落研究的重要参数,反映植物群落的生长状况及其所处生存环境的优劣。小尺度常规的测定方法费力、费时,而且是破坏性的,不能动态监测其变化。而对于大尺度的测定,常规方法无能为力,只能采用遥感方法。应用人工神经网络和多谱段遥感数据对香港大屿山岛进行土地覆盖的分类,设计了一个合适的多层感知器前向反馈神经网络用于土地覆盖分类,并将分类结果与传统的最大似然分类方法所得的结果作比较,结果表明神经网络方法在分类精度上有了很大的提高。     

Fuzzy regression modeling for tool performance prediction and degradation detection

In this paper, the viability of using Fuzzy-Rule-Based Regression Modeling (FRM) algorithm for tool performance and degradation detection is investigated. The FRM is developed based on a multi-layered fuzzy-rule-based hybrid system with Multiple Regression Models (MRM) embedded into a fuzzy logic inference engine that employs Self Organizing Maps (SOM) for clustering. The FRM converts a complex nonlinear problem to a simplified linear format in order to further increase the accuracy in prediction and rate of convergence. The efficacy of the proposed FRM is tested through a case study - namely to predict the remaining useful life of a ball nose milling cutter during a dry machining process of hardened tool steel with a hardness of 52-54 HRc. A comparative study is further made between four predictive models using the same set of experimental data. It is shown that the FRM is superior as compared with conventional MRM, Back Propagation Neural Networks (BPNN) and Radial Basis Function Networks (RBFN) in terms of prediction accuracy and learning speed.     

Supervised Neural Q_learning based Motion Control for Bionic Underwater Robots

Bionic underwater robots have been a hot research area in recent years. The motion control methods for a kind of bionic underwater robot with two undulating fins are discussed in this paper. The equations of motion for the bionic underwater robot are described. To apply the reinforcement learning to the actual robot control, a Supervised Neural Q_learning (SNQL) algorithm is put forward. This algorithm is based on conventional Q_learning algorithm, but has three remarkable distinctions: (1) using a feedforward neural network to approximate the Q_function table; (2) adopting a learning sample database to speed up learning and improve the stability of learning system; (3) introducing a supervised control in the earlier stage of learning for safety and to speed up learning again. Experiments of swimming straightforward are carried out with SNQL algorithm. Results indicate that the SNQL algorithm is more effective than pure neural Q_learning or supervised control. It is a feasible approach to figure out the motion control for bionic underwater robots.     

Eye tracking using artificial neural networks for human computer interaction

This paper describes an ongoing project that has the aim to develop a low cost application to replace a computer mouse for people with physical impairment. The application is based on an eye tracking algorithm and assumes that the camera and the head position are fixed. Color tracking and template matching methods are used for pupil detection. Calibration is provided by neural networks as well as by parametric interpolation methods. Neural networks use back-propagation for learning and bipolar sigmoid function is chosen as the activation function. The user's eye is scanned with a simple web camera with backlight compensation which is attached to a head fixation device. Neural networks significantly outperform parametric interpolation techniques: 1) the calibration procedure is faster as they require less calibration marks and 2) cursor control is more precise. The system in its current stage of development is able to distinguish regions at least on the level of desktop icons. The main limitation of the proposed method is the lack of head-pose invariance and its relative sensitivity to illumination (especially to incidental pupil reflections).     

Automating parallel implementation of neural learning algorithms

Neural learning algorithms generally involve a number of identical processing units, which are fully or partially connected, and involve an update function, such as a ramp, a sigmoid or a Gaussian function for instance. Some variations also exist, where units can be heterogeneous, or where an alternative update technique is employed, such as a pulse stream generator. Associated with connections are numerical values that must be adjusted using a learning rule, and and dictated by parameters that are learning rule specific, such as momentum, a learning rate, a temperature, amongst others. Usually, neural learning algorithms involve local updates, and a global interaction between units is often discouraged, except in instances where units are fully connected, or involve synchronous updates. In all of these instances, concurrency within a neural algorithm cannot be fully exploited without a suitable implementation strategy. A design scheme is described for translating a neural learning algorithm from inception to implementation on a parallel machine using PVM or MPI libraries, or onto programmable logic such as FPGAs. A designer must first describe the algorithm using a specialised Neural Language, from which a Petri net (PN) model is constructed automatically for verification, and building a performance model. The PN model can be used to study issues such as synchronisation points, resource sharing and concurrency within a learning rule. Specialised constructs are provided to enable a designer to express various aspects of a learning rule, such as the number and connectivity of neural nodes, the interconnection strategies, and information flows required by the learning algorithm. A scheduling and mapping strategy is then used to translate this PN model onto a multiprocessor template. We demonstrate our technique using a Kohonen and backpropagation learning rules, implemented on a loosely coupled workstation cluster, and a dedicated parallel machine, with PVM libraries.     

Binary neural network training algorithms based on linear sequential learning

A key problem in Binary Neural Network learning is to decide bigger linear separable subsets. In this paper we prove some lemmas about linear separability. Based on these lemmas, we propose Multi-Core Learning (MCL) and Multi-Core Expand-and-Truncate Learning (MCETL) algorithms to construct Binary Neural Networks. We conclude that MCL and MCETL simplify the equations to compute weights and thresholds, and they result in the construction of simpler hidden layer. Examples are given to demonstrate these conclusions.