Constructing a query-able radial basis function artificial neural network

Artificial neural networks will be more widely accepted as standard engineering tools if their reasoning process can be made less opaque. This paper describes NetQuery, an explanation mechanism that extracts meaningful explanations from trained Radial Basis Function (RBF) networks. RBF networks are well suited for explanation generation because they contain a set of locally tuned units. Standard RBF networks are modified to identify dependencies between the inputs, to be sparsely connected, and to have an easily interpretable output layer. Given these modifications, the network architecture can be used to extract "Why?" and "Why not?" explanations from the network in terms of excitatory and inhibitory in-puts and their linear relationships, greatly simplified by a run-time pruning algorithm. These query results are validated by creating an expert system based on the explanations. NetQuery is also able to inform a user about a possible change in category for a given pattern by responding to a "How can I...?" query. This kind of query is extremely useful when analyzing the quality of a pattern set.     

Evaluation of soil aggregate stability in Algerian northwestern soils using pedotransfer functions and artificial neural networks

Since, aggregate stability is the main physical property regulating erodibility; its observations can act as a useful indicator for monitoring and managing soil degradation. In this context, this study carried out in the alluvial plain of Cheliff, a semi-arid area aimed to predict aggregate stability through Mean Weight Diameter (MWD), using pedotransfer functions (PTFs) with different stratifications (textural, salinity and organic-textural) and artificial neural networks (ANNs). Results showed that the best MWD predictions were those related to organic-textural PTFs, in this stratification the silty-clay moderately rich OM class showed the highest significant determination coefficient R² (0.65) and the lowest mean square error (0.03), whereas, the textural and salinity PTFs were a very weak predictors with a very low R². It was also found that the performances of ANNs in predicting MWD were better than those of PTFs, regarding ANNs input variables the best predictions were those obtained with a large number of input variables, furthermore, by using a large number of hidden neurons, the performances of Radial Basis Function (RBF) were better than those of Multilayer Perceptron (MLP). It was also noted that the best RBF results were always related to the Gaussian hidden activation, whereas, MLP was not related to a specific hidden activation.     

Short-term prediction of chaotic time series by using RBF network with regression weights

We propose a framework for constructing and training a radial basis function (RBF) neural network. The structure of the gaussian functions is modified using a pseudo-gaussian function (PG) in which two scaling parameters sigma are introduced, which eliminates the symmetry restriction and provides the neurons in the hidden layer with greater flexibility with respect to function approximation. We propose a modified PG-BF (pseudo-gaussian basis function) network in which the regression weights are used to replace the constant weights in the output layer. For this purpose, a sequential learning algorithm is presented to adapt the structure of the network, in which it is possible to create a new hidden unit and also to detect and remove inactive units. A salient feature of the network systems is that the method used for calculating the overall output is the weighted average of the output associated with each receptive field. The superior performance of the proposed PG-BF system over the standard RBF are illustrated using the problem of short-term prediction of chaotic time series.     

Cooperative coevolution of neural representations

A genetic algorithm (GA) is used to search for a set of local feature detectors or hidden units. These are in turn employed as a representation of the input data for neural learning in the upper layer of a multilayer perceptron (MLP) which performs an image classification task. Three different methods of encoding hidden unit weights in the chromosome of the GA are presented, including one which coevolves all the feature detectors in a single chromosome, and two which promote the cooperation of feature detectors by encoding them in their own individual chromosomes. The fitness function measures the MLP classification accuracy together with the confidence of the networks.     

Predicting spatial data with RBF networks

Spatial prediction needs to account for spatial information, which makes conventional radial basis function (RBF) networks inappropriate, for they assume independent and identical distribution. In this paper, we fuse spatial information at different layers of RBF. Experiments show fusion at hidden layer gives the best result and suggest that the optimal value is around one for the coefficient, which is used in the linear combination at the output layer.     

Automatic determination of radial basis functions: an immunity-based approach

The appropriate operation of a radial basis function (RBF) neural network depends mainly upon an adequate choice of the parameters of its basis functions. The simplest approach to train an RBF network is to assume fixed radial basis functions defining the activation of the hidden units. Once the RBF parameters are fixed, the optimal set of output weights can be determined straightforwardly by using a linear least squares algorithm, which generally means reduction in the learning time as compared to the determination of all RBF network parameters using supervised learning. The main drawback of this strategy is the requirement of an efficient algorithm to determine the number, position, and dispersion of the RBFs. The approach proposed here is inspired by models derived from the vertebrate immune system, that will be shown to perform unsupervised cluster analysis. The algorithm is introduced and its performance is compared to that of the random, k-means center selection procedures and other results from the literature. By automatically defining the number of RBF centers, their positions and dispersions, the proposed method leads to parsimonious solutions. Simulation results are reported concerning regression and classification problems.     

EEG source localization: comparative study of classical and neural network methods.

We address the problem of estimating biopotential sources within the brain, based on EEG signals observed on the scalp. This problem, known as the inverse problem of electrophysiology, has no closed-form solution, and requires iterative techniques such as the Levenberg-Marquardt (LM) algorithm. Considering the nonlinear nature of the inverse problem, and the low signal to noise ratio inherent in EEG signals, a backpropagation neural network (BPN) has been recently proposed as a solution. The technique has not been properly compared with classical techniques such as the LM method, or with more recent neural network techniques such as the Radial Basis Function (RBF) network. In this paper, we provide improved strategies based on BPN and consider RBF networks in solving the inverse problem. We compare the performances of BPN, RBF and a hybrid technique with that of the classical LM method.     

Variable selection in nonlinear modeling based on RBF networks and evolutionary computation

In this paper a novel variable selection method based on Radial Basis Function (RBF) neural networks and genetic algorithms is presented. The fuzzy means algorithm is utilized as the training method for the RBF networks, due to its inherent speed, the deterministic approach of selecting the hidden node centers and the fact that it involves only a single tuning parameter. The trade-off between the accuracy and parsimony of the produced model is handled by using Final Prediction Error criterion, based on the RBF training and validation errors, as a fitness function of the proposed genetic algorithm. The tuning parameter required by the fuzzy means algorithm is treated as a free variable by the genetic algorithm. The proposed method was tested in benchmark data sets stemming from the scientific communities of time-series prediction and medicinal chemistry and produced promising results.     

Prediction of the disulfide bonding state of cysteines in proteins with hidden neural networks

A hybrid system (hidden neural network) based on a hidden Markov model (HMM) and neural networks (NN) was trained to predict the bonding states of cysteines in proteins starting from the residue chains. Training was performed using 4136 cysteine-containing segments extracted from 969 non-homologous proteins of well-resolved 3D structure and without chain-breaks. After a 20-fold cross-validation procedure, the efficiency of the prediction scores as high as 80% using neural networks based on evolutionary information. When the whole protein is taken into account by means of an HMM, a hybrid system is generated, whose emission probabilities are computed using the NN output (hidden neural networks). In this case, the predictor accuracy increases up to 88%. Further, when tested on a protein basis, the hybrid system can correctly predict 84% of the chains in the data set, with a gain of at least 27% over the NN predictor.     

Development of simple fitness landscapes for peptides by artificial neural filter systems

The applicability of artificial neural filter systems as fitness functions for sequence-oriented peptide design was evaluated. Two example applications were selected: classification of dipeptides according to their hydrophobicity and classification of proteolytic cleavage-sites of protein precursor sequences according to their mean hydrophobicities and mean side-chain volumes. The cleavage-sites covered 12 residues. In the dipeptide experiments the objective was to separate a selected set of molecules from all other possible dipeptide sequences. Perceptrons, feedforward networks with one hidden layer, and a hybrid network were applied. The filters were trained by a (1,) evolution strategy. Two types of network units employing either a sigmoidal or a unimodal transfer function were used in the feedforward filters, and their influence on classification was investigated. The two-layer hybrid network employed gaussian activation functions. To analyze classification of the different filter systems, their output was plotted in the two-dimensional sequence space. The diagrams were interpreted as fitness landscapes qualifying the markedness of a characteristic peptide feature which can be used as a guide through sequence space for rational peptide design. It is demonstrated that the applicability of neural filter systems as a heuristic method for sequence optimization depends on both the appropriate network architecture and selection of representative sequence data. The networks with unimodal activation functions and the hybrid networks both led to a number of local optima. However, the hybrid networks produced the best prediction results. In contrast, the filters with sigmoidal activation produced good reclassification results leading to fitness landscapes lacking unreasonable local optima. Similar results were obtained for classification of both dipeptides and cleavage-site sequences.     

Predictive assessment of ochratoxin A accumulation in grape juice based-medium by Aspergillus carbonarius using neural networks

Aims:  To study the ability of multi-layer perceptron artificial neural networks (MLP-ANN) and radial-basis function networks (RBFNs) to predict ochratoxin A (OTA) concentration over time in grape-based cultures of Aspergillus carbonarius under different conditions of temperature, water activity ( a _w) and sub-inhibitory doses of the fungicide carbendazim.
Methods and Results:  A strain of A. carbonarius was cultured in a red grape juice-based medium. The input variables to the network were temperature (20–28°C), a _w (0·94–0·98), carbendazim level (0–450 ng ml⁻¹) and time (3–15 days after the lag phase). The output of the ANNs was OTA level determined by liquid chromatography. Three algorithms were comparatively tested for MLP. The lowest error was obtained by MLP without validation. Performance decreased when hold-out validation was accomplished but the risk of over-fitting is also lower. The best MLP architecture was determined. RBFNs provided similar performances but a substantially higher number of hidden nodes were needed.
Conclusions:  ANNs are useful to predict OTA level in grape juice cultures of A. carbonarius over a range of a _w, temperature and carbendazim doses.
Significance and Impact of the Study:  This is a pioneering study on the application of ANNs to forecast OTA accumulation in food based substrates. These models can be similarly applied to other mycotoxins and fungal species.     

A novel hybrid GA/RBFNN technique for protein sequences classification

A novel hybrid genetic algorithm (GA)/radial basis function neural network (RBFNN) technique, which selects features from the protein sequences and trains the RBF neural network simultaneously, is proposed in this paper. Experimental results show that the proposed hybrid GA/RBFNN system outperforms the BLAST and the HMMer.     

On the structure and initial parameter identification of Gaussian RBF networks

We consider the efficient initialization of structure and parameters of generalized Gaussian radial basis function (RBF) networks using fuzzy decision trees generated by fuzzy ID3 like induction algorithms. The initialization scheme is based on the proposed functional equivalence property of fuzzy decision trees and generalized Gaussian RBF networks. The resulting RBF network is compact, easy to induce, comprehensible, and has acceptable classification accuracy with stochastic gradient descent learning algorithm.     

Complex-valued minimal resource allocation network for nonlinear signal processing

This paper presents a sequential learning algorithm and evaluates its performance on complex valued signal processing problems. The algorithm is referred to as Complex Minimal Resource Allocation Network (CMRAN) algorithm and it is an extension of the MRAN algorithm originally developed for online learning in real valued RBF networks. CMRAN has the ability to grow and prune the (complex) RBF network's hidden neurons to ensure a parsimonious network structure. The performance of the learning algorithm is illustrated using two applications from signal processing of communication systems. The first application considers identification of a nonlinear complex channel. The second application considers the use of CMRAN to QAM digital channel equalization problems. Simulation results presented clearly show that CMRAN is very effective in modeling and equalization with performance achieved often being superior to that of some of the well known methods.     

NF-RCNN: Heart localization and right ventricle wall motion abnormality detection in cardiac MRI

Convolutional neural networks (CNNs) are extensively used in cardiac image analysis. However, heart localization has become a prerequisite to these networks since it decreases the size of input images. Accordingly, recent CNNs benefit from deeper architectures in gaining abstract semantic information. In the present study, a deep learning-based method was developed for heart localization in cardiac MR images. Further, Network in Network (NIN) was used as the region proposal network (RPN) of the faster R-CNN, and then NIN Faster-RCNN (NF-RCNN) was proposed. NIN architecture is formed based on “MLPCONV” layer, a combination of convolutional network and multilayer perceptron (MLP). Therefore, it could deal with the complicated structures of MR images. Furthermore, two sets of cardiac MRI dataset were used to evaluate the network, and all the evaluation metrics indicated an absolute superiority of the proposed network over all related networks. In addition, FROC curve, precision-recall (PR) analysis, and mean localization error were employed to evaluate the proposed network. In brief, the results included an AUC value of 0.98 for FROC curve, a mean average precision of 0.96 for precision-recall curve, and a mean localization error of 6.17 mm. Moreover, a deep learning-based approach for the right ventricle wall motion analysis (WMA) was performed on the first dataset and the effect of the heart localization on this algorithm was studied. The results revealed that NF-RCNN increased the speed and decreased the required memory significantly.     

Use of neural networks in the mathematical modelling of the enzymic synthesis of amoxicillin catalysed by penicillin G acylase immobilized in chitosan

This study focuses in the mathematical modelling of the enzymic synthesis of amoxicillin by the reaction of p-hydroxyphenylglycine methyl ester and 6-aminopenicillanic acid (6APA), catalyzed by penicillin G acylase (PGA) immobilized on glutaraldehyde-chitosan, at 25°C and pH 6.5. Previous work on the kinetics and mechanism of reaction showed that the use of neural networks seems to be an interesting alternative to simulate experimental data of antibiotic production. Therefore, two feedforward neural networks, with one hidden layer, were trained and used to forecast the rates of amoxicillin and p-hydroxyphenylglycine (POHPG) net production. First of all, some parameters that affect the network performed were investigated, such as the number of nodes between the input and hidden layers and the number of interactions during the learning phase. Afterwards, hybrid models that coupled artificial neural networks to mass-balance equations were used to reproduce the performance of batch reactors for the production of amoxicillin. This approach provided accurate results, within the range of substrate concentration studied.     

An improvement of extreme learning machine for compact single-hidden-layer feedforward neural networks

Recently, a novel learning algorithm called extreme learning machine (ELM) was proposed for efficiently training single-hidden-layer feedforward neural networks (SLFNs). It was much faster than the traditional gradient-descent-based learning algorithms due to the analytical determination of output weights with the random choice of input weights and hidden layer biases. However, this algorithm often requires a large number of hidden units and thus slowly responds to new observations. Evolutionary extreme learning machine (E-ELM) was proposed to overcome this problem; it used the differential evolution algorithm to select the input weights and hidden layer biases. However, this algorithm required much time for searching optimal parameters with iterative processes and was not suitable for data sets with a large number of input features. In this paper, a new approach for training SLFNs is proposed, in which the input weights and biases of hidden units are determined based on a fast regularized least-squares scheme. Experimental results for many real applications with both small and large number of input features show that our proposed approach can achieve good generalization performance with much more compact networks and extremely high speed for both learning and testing.     

Pruning artificial neural networks using neural complexity measures

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.     

Bayesian Neural Network Models for Censored Data

Neural networks are considered by many to be very promising tools for classification and prediction. The flexibility of the neural network models often result in over-fit. Shrinking the parameters using a penalized likelihood is often used in order to overcome such over-fit. In this paper we extend the approach proposed by FARAGGI and SIMON (1995a) to modeling censored survival data using the input-output relationship associated with a single hidden layer feed-forward neural network. Instead of estimating the neural network parameters using the method of maximum likelihood, we place normal prior distributions on the parameters and make inferences based on derived posterior distributions of the parameters. This Bayesian formulation will result in shrinking the parameters of the neural network model and will reduce the over-fit compared with the maximum likelihood estimators. We illustrate our proposed method on a simulated and a real example.     

A new approach for categorizing pig lying behaviour based on a Delaunay triangulation method

Machine vision-based monitoring of pig lying behaviour is a fast and non-intrusive approach that could be used to improve animal health and welfare. Four pens with 22 pigs in each were selected at a commercial pig farm and monitored for 15 days using top view cameras. Three thermal categories were selected relative to room setpoint temperature. An image processing technique based on Delaunay triangulation (DT) was utilized. Different lying patterns (close, normal and far) were defined regarding the perimeter of each DT triangle and the percentages of each lying pattern were obtained in each thermal category. A method using a multilayer perceptron (MLP) neural network, to automatically classify group lying behaviour of pigs into three thermal categories, was developed and tested for its feasibility. The DT features (mean value of perimeters, maximum and minimum length of sides of triangles) were calculated as inputs for the MLP classifier. The network was trained, validated and tested and the results revealed that MLP could classify lying features into the three thermal categories with high overall accuracy (95.6%). The technique indicates that a combination of image processing, MLP classification and mathematical modelling can be used as a precise method for quantifying pig lying behaviour in welfare investigations.