Artificial neural networks for blur identification and restoration of nonlinearly degraded images.

In this paper, an image restoration algorithm is proposed to identify noncausal blur function. Image degradation processes include both linear and nonlinear phenomena. A neural network model combining an adaptive auto-associative network with a random Gaussian process is proposed to restore the blurred image and blur function simultaneously. The noisy and blurred images are modeled as continuous associative networks, whereas auto-associative part determines the image model coefficients and the hetero-associative part determines the blur function of the system. The self-organization like structure provides the potential solution of the blind image restoration problem. The estimation and restoration are implemented by using an iterative gradient based algorithm to minimize the error function.     

Elements for a general memory structure: properties of recurrent neural networks used to form situation models

We study how individual memory items are stored assuming that situations given in the environment can be represented in the form of synaptic-like couplings in recurrent neural networks. Previous numerical investigations have shown that specific architectures based on suppression or max units can successfully learn static or dynamic stimuli (situations). Here we provide a theoretical basis concerning the learning process convergence and the network response to a novel stimulus. We show that, besides learning “simple” static situations, a nD network can learn and replicate a sequence of up to n different vectors or frames. We find limits on the learning rate and show coupling matrices developing during training in different cases including expansion of the network into the case of nonlinear interunit coupling. Furthermore, we show that a specific coupling matrix provides low-pass-filter properties to the units, thus connecting networks constructed by static summation units with continuous-time networks. We also show under which conditions such networks can be used to perform arithmetic calculations by means of pattern completion.     

Self-organization of associative memory and pattern classification: recurrent signal processing on topological feature maps

We extend the neural concepts of topological feature maps towards self-organization of auto-associative memory and hierarchical pattern classification. As is well-known, topological maps for statistical data sets store information on the associated probability densities. To extract that information we introduce a recurrent dynamics of signal processing. We show that the dynamics converts a topological map into an auto-associative memory for real-valued feature vectors which is capable to perform a cluster analysis. The neural network scheme thus developed represents a generalization of non-linear matrix-type associative memories. The results naturally lead to the concept of a feature atlas and an associated scheme of self-organized, hierarchical pattern classification.     

Analytical solution to the coupled evolution of multidimensional NMR data

A substantial time savings in the collection of multidimensional NMR data can be achieved by coupling the evolution of nuclei in the indirect dimensions. In order to save time, the sampling of the indirect dimensions is inherently incomplete. Therefore, many algorithms and samplings schemes have been developed aimed at separating the coevolved frequencies into analyzable data with limited artifacts. This paper extends the use of circulant matrices to describe coupled evolution with convolutions. By understanding the data in terms of convolutions, there is an exact solution to the inversion problem of extracting the orthogonal vectors from the coupled dimensions. Previously, this inversion problem has been solved using peak coordinates extracted from spectra. In contrast, the method described here uses spectra directly. This solution suggests a simple sampling scheme of collecting N orthogonal spectra, and N + 1 projections at specific projection angles, however, the theory developed can be extended generally to arbitrary projection angles. The circulant matrix methodology is demonstrated for simulated and real data. Further, an algorithm for separating overlapped signals in the detected dimension is presented. The algorithm involves the forward calculation of the coupled spectra from the orthogonal spectra, followed by back calculation of the orthogonal spectra from the coupled spectra, thus permitting rigorous cross-validation. This algorithm is shown to be robust in that erroneous solutions give rise to large artifacts. Electronic supplementary material  The online version of this article (doi:) contains supplementary material, which is available to authorized users.     

Fast adaptive formation of orthogonalizing filters and associative memory in recurrent networks of neuron-like elements

A demonstration is given that an orthogonalizing filter for patterns is formed adaptively and very rapidly in a network of neuron-like elements with internal feedback connections. It is here assumed that the feedback gain is variable, and proportional to the correlation matrix of the output pattern vectors. The time-dependent signal transfer properties of the complete system are described by a system matrix which satisfies a matrix Bernoulli differential equation; solutions of this equation are outlined. The asymptotic value of the system matrix is shown to correspond to the orthogonal projection operator on the space that is complementary to the space spanned by all of the earlier input pattern vectors. Such a system then acts as a filter, which optimally extracts the amount that is new in an input pattern with respect to all old patterns. It also has features that are directly attributable to a distributed associative memory that is optimally selective.     

A New Approach for Inversion of Large Random Matrices in Massive MIMO Systems

We report a novel approach for inversion of large random matrices in massive Multiple-Input Multiple Output (MIMO) systems. It is based on the concept of inverse vectors in which an inverse vector is defined for each column of the principal matrix. Such an inverse vector has to satisfy two constraints. Firstly, it has to be in the null-space of all the remaining columns. We call it the null-space problem. Secondly, it has to form a projection of value equal to one in the direction of selected column. We term it as the normalization problem. The process essentially decomposes the inversion problem and distributes it over columns. Each column can be thought of as a node in the network or a particle in a swarm seeking its own solution, the inverse vector, which lightens the computational load on it. Another benefit of this approach is its applicability to all three cases pertaining to a linear system: the fully-determined, the over-determined, and the under-determined case. It eliminates the need of forming the generalized inverse for the last two cases by providing a new way to solve the least squares problem and the Moore and Penrose''s pseudoinverse problem. The approach makes no assumption regarding the size, structure or sparsity of the matrix. This makes it fully applicable to much in vogue large random matrices arising in massive MIMO systems. Also, the null-space problem opens the door for a plethora of methods available in literature for null-space computation to enter the realm of matrix inversion. There is even a flexibility of finding an exact or approximate inverse depending on the null-space method employed. We employ the Householder''s null-space method for exact solution and present a complete exposition of the new approach. A detailed comparison with well-established matrix inversion methods in literature is also given.     

Corrected small-sample estimation of the Bayes error

MOTIVATION: A major problem of pattern classification is estimation of the Bayes error when only small samples are available. One way to estimate the Bayes error is to design a classifier based on some classification rule applied to sample data, estimate the error of the designed classifier, and then use this estimate as an estimate of the Bayes error. Relative to the Bayes error, the expected error of the designed classifier is biased high, and this bias can be severe with small samples. RESULTS: This paper provides a correction for the bias by subtracting a term derived from the representation of the estimation error. It does so for Boolean classifiers, these being defined on binary features. Although the general theory applies to any Boolean classifier, a model is introduced to reduce the number of parameters. A key point is that the expected correction is conservative. Properties of the corrected estimate are studied via simulation. The correction applies to binary predictors because they are mathematically identical to Boolean classifiers. In this context the correction is adapted to the coefficient of determination, which has been used to measure nonlinear multivariate relations between genes and design genetic regulatory networks. An application using gene-expression data from a microarray experiment is provided on the website http://gspsnap.tamu.edu/smallsample/ (user:'smallsample', password:'smallsample)').     

Mathematical aspects of outer-product asynchronous content-addressable memories

Conclusions We have given evidence by mathematical analysis and by example that the construction of CAM's as quasineural networks based on the twin principles of an outer-product weight matrix and a random asynchronous single-neuron dynamics encounters two obstacles to good performance which appear to be inherent. It is desirable to have the stored memory vectors of a CAM as mutually far apart as possible in order to have unambiguous retrieval with as large a fraction of initial errors (minrad) as possible. For a given number,m, of memory vectors to be stored this requires that their dimension,n, be larger thanm and that the memory vectors be nearly mutually orthogonal (except for complementary pairs). In HOCAM's, it does not seem possible to have both a large minrad and an efficient ratiom/n. Attempts to increasem/n are likely to introduce extraneous fixed-points which reduce minrad appreciably. We have demonstrated this phenomenon in several cases for a particular mode of constructing CAM's of arbitrary size which have a desirable spacing between memory vectors. We conjecture that it is present also in HOCAM's having a random selection of memory vectors. (A mathematical proof of this conjecture now seems possible.) This may account for the rather pessimistic results on capacity obtained by mathematical analysis here and in Cottrell (1988), by a probabilistic analysis in Posner (1987) and by simulation in Hopfield (1982). Further, in Cottrell (1988) there is evidence that outer-product weights are near optimal with respect to minrad, so that otherW may not improve matters.We have left to another paper a study of other approaches to content-addressable memories of which we are aware, but which are not focused on asynchronous dynamics; e.g. computer CAM's as in Kohonen (1977) and biological memory models as in Little (1974); Palm (1980) and Little and Shaw (1978). We have not considered the learning, or adaptive, aspects of CAM's. However, insofar as learning is Hebbian and leads to outer-product weights, our analysis has implications for the effectiveness of learned weights, as may be inferred from our results on ambiguous retrieval.This research was partially supported by NSF Grant CCR-87121192 and AFOSR Grant 88-0245     

Topological maps of protein sequences

A new method based on neural networks to cluster proteins into families is described. The network is trained with the Kohonen unsupervised learning algorithm, using matrix pattern representations of the protein sequences as inputs. The components (x, y) of these 20×20 matrix patterns are the normalized frequencies of all pairs xy of amino acids in each sequence. We investigate the influence of different learning parameters in the final topological maps obtained with a learning set of ten proteins belonging to three established families. In all cases, except in those where the synaptic vectors remains nearly unchanged during learning, the ten proteins are correctly classified into the expected families. The classification by the trained network of mutated or incomplete sequences of the learned proteins is also analysed. The neural network gives a correct classification for a sequence mutated in 21.5%±7% of its amino acids and for fragments representing 7.5%±3% of the original sequence. Similar results were obtained with a learning set of 32 proteins belonging to 15 families. These results show that a neural network can be trained following the Kohonen algorithm to obtain topological maps of protein sequences, where related proteins are finally associated to the same winner neuron or to neighboring ones, and that the trained network can be applied to rapidly classify new sequences. This approach opens new possibilities to find rapid and efficient algorithms to organize and search for homologies in the whole protein database.     

Non-linear data structure extraction using simple hebbian networks

. We present a class a neural networks algorithms based on simple hebbian learning which allow the finding of higher order structure in data. The neural networks use negative feedback of activation to self-organise; such networks have previously been shown to be capable of performing principal component analysis (PCA). In this paper, this is extended to exploratory projection pursuit (EPP), which is a statistical method for investigating structure in high-dimensional data sets. As opposed to previous proposals for networks which learn using hebbian learning, no explicit weight normalisation, decay or weight clipping is required. The results are extended to multiple units and related to both the statistical literature on EPP and the neural network literature on non-linear PCA. Received: 30 May 1994/Accepted in revised form: 18 November 1994     

Creative dynamics approach to neural intelligence

The thrust of this paper is to introduce and discuss a substantially new type of dynamical system for modelling biological behavior. The approach was motivated by an attempt to remove one of the most fundamental limitations of artificial neural networks — their rigid behavior compared with even simplest biological systems. This approach exploits a novel paradigm in nonlinear dynamics based upon the concept of terminal attractors and repellers. It was demonstrated that non-Lipschitzian dynamics based upon the failure of Lipschitz condition exhibits a new qualitative effect — a multi-choice response to periodic external excitations. Based upon this property, a substantially new class of dynamical systems — the unpredictable systems — was introduced and analyzed. These systems are represented in the form of coupled activation and learning dynamical equations whose ability to be spontaneously activated is based upon two pathological characteristics. Firstly, such systems have zero Jacobian. As a result of that, they have an infinite number of equilibrium points which occupy curves, surfaces or hypersurfaces. Secondly, at all these equilibrium points, the Lipschitz conditions fails, so the equilibrium points become terminal attractors or repellers depending upon the sign of the periodic excitation. Both of these pathological characteristics result in multi-choice response of unpredictable dynamical systems. It has been shown that the unpredictable systems can be controlled by sign strings which uniquely define the system behaviors by specifying the direction of the motions in the critical points. By changing the combinations of signs in the code strings the system can reproduce any prescribed behavior to a prescribed accuracy. That is why the unpredictable systems driven by sign strings are extremely flexible and are highly adaptable to environmental changes. It was also shown that such systems can serve as a powerful tool for temporal pattern memories and complex pattern recognition. It has been demonstrated that new architecture of neural networks based upon non-Lipschitzian dynamics can be utilized for modelling more complex patterns of behavior which can be associated with phenomenological models of creativity and neural intelligence.     

Memory Storage Fidelity in the Hippocampal Circuit: The Role of Subregions and Input Statistics

In the last decades a standard model regarding the function of the hippocampus in memory formation has been established and tested computationally. It has been argued that the CA3 region works as an auto-associative memory and that its recurrent fibers are the actual storing place of the memories. Furthermore, to work properly CA3 requires memory patterns that are mutually uncorrelated. It has been suggested that the dentate gyrus orthogonalizes the patterns before storage, a process known as pattern separation. In this study we review the model when random input patterns are presented for storage and investigate whether it is capable of storing patterns of more realistic entorhinal grid cell input. Surprisingly, we find that an auto-associative CA3 net is redundant for random inputs up to moderate noise levels and is only beneficial at high noise levels. When grid cell input is presented, auto-association is even harmful for memory performance at all levels. Furthermore, we find that Hebbian learning in the dentate gyrus does not support its function as a pattern separator. These findings challenge the standard framework and support an alternative view where the simpler EC-CA1-EC network is sufficient for memory storage.     

An Efficient Supervised Training Algorithm for Multilayer Spiking Neural Networks

The spiking neural networks (SNNs) are the third generation of neural networks and perform remarkably well in cognitive tasks such as pattern recognition. The spike emitting and information processing mechanisms found in biological cognitive systems motivate the application of the hierarchical structure and temporal encoding mechanism in spiking neural networks, which have exhibited strong computational capability. However, the hierarchical structure and temporal encoding approach require neurons to process information serially in space and time respectively, which reduce the training efficiency significantly. For training the hierarchical SNNs, most existing methods are based on the traditional back-propagation algorithm, inheriting its drawbacks of the gradient diffusion and the sensitivity on parameters. To keep the powerful computation capability of the hierarchical structure and temporal encoding mechanism, but to overcome the low efficiency of the existing algorithms, a new training algorithm, the Normalized Spiking Error Back Propagation (NSEBP) is proposed in this paper. In the feedforward calculation, the output spike times are calculated by solving the quadratic function in the spike response model instead of detecting postsynaptic voltage states at all time points in traditional algorithms. Besides, in the feedback weight modification, the computational error is propagated to previous layers by the presynaptic spike jitter instead of the gradient decent rule, which realizes the layer-wised training. Furthermore, our algorithm investigates the mathematical relation between the weight variation and voltage error change, which makes the normalization in the weight modification applicable. Adopting these strategies, our algorithm outperforms the traditional SNN multi-layer algorithms in terms of learning efficiency and parameter sensitivity, that are also demonstrated by the comprehensive experimental results in this paper.     

Pattern storage in a sparsely coded neural network with cyclic activation

We investigate an artificial neural network model with a modified Hebb rule. It is an auto-associative neural network similar to the Hopfield model and to the Willshaw model. It has properties of both of these models. Another property is that the patterns are sparsely coded and are stored in cycles of synchronous neural activities. The cycles of activity for some ranges of parameter increase the capacity of the model. We discuss basic properties of the model and some of the implementation issues, namely optimizing of the algorithms. We describe the modification of the Hebb learning rule, the learning algorithm, the generation of patterns, decomposition of patterns into cycles and pattern recall.     

A Low Dimensional Approach on Network Characterization

In many applications, one may need to characterize a given network among a large set of base networks, and these networks are large in size and diverse in structure over the search space. In addition, the characterization algorithms are required to have low volatility and with a small circle of uncertainty. For large datasets, these algorithms are computationally intensive and inefficient. However, under the context of network mining, a major concern of some applications is speed. Hence, we are motivated to develop a fast characterization algorithm, which can be used to quickly construct a graph space for analysis purpose. Our approach is to transform a network characterization measure, commonly formulated based on similarity matrices, into simple vector form signatures. We shall show that the similarity matrix can be represented by a dyadic product of two N-dimensional signature vectors; thus the network alignment process, which is usually solved as an assignment problem, can be reduced into a simple alignment problem based on separate signature vectors.     

A modular latching chain

Many cognitive tasks involve transitions between distinct mental processes, which may range from discrete states to complex strategies. The ability of cortical networks to combine discrete jumps with continuous glides along ever changing trajectories, dubbed latching dynamics, may be essential for the emergence of the unique cognitive capacities of modern humans. Novel trajectories have to be followed in the multidimensional space of cortical activity for novel behaviours to be produced; yet, not everything changes: several lines of evidence point at recurring patterns in the sequence of activation of cortical areas in a variety of behaviours. To extend a mathematical model of latching dynamics beyond the simple unstructured auto-associative Potts network previously analysed, we introduce delayed structured connectivity and hetero-associative connection weights, and we explore their effects on the dynamics. A modular model in the small-world regime is considered, with modules arranged on a ring. The synaptic weights include a standard auto-associative component, stabilizing distinct patterns of activity, and a hetero-associative component, favoring transitions from one pattern, expressed in one module, to the next, in the next module. We then study, through simulations, how structural parameters, like those regulating rewiring probability, noise and feedback connections, determine sequential association dynamics.     

CD-Based Indices for Link Prediction in Complex Network

Lots of similarity-based algorithms have been designed to deal with the problem of link prediction in the past decade. In order to improve prediction accuracy, a novel cosine similarity index CD based on distance between nodes and cosine value between vectors is proposed in this paper. Firstly, node coordinate matrix can be obtained by node distances which are different from distance matrix and row vectors of the matrix are regarded as coordinates of nodes. Then, cosine value between node coordinates is used as their similarity index. A local community density index LD is also proposed. Then, a series of CD-based indices include CD-LD-k, CD*LD-k, CD-k and CDI are presented and applied in ten real networks. Experimental results demonstrate the effectiveness of CD-based indices. The effects of network clustering coefficient and assortative coefficient on prediction accuracy of indices are analyzed. CD-LD-k and CD*LD-k can improve prediction accuracy without considering the assortative coefficient of network is negative or positive. According to analysis of relative precision of each method on each network, CD-LD-k and CD*LD-k indices have excellent average performance and robustness. CD and CD-k indices perform better on positive assortative networks than on negative assortative networks. For negative assortative networks, we improve and refine CD index, referred as CDI index, combining the advantages of CD index and evolutionary mechanism of the network model BA. Experimental results reveal that CDI index can increase prediction accuracy of CD on negative assortative networks.