Encoding time in neural dynamic regimes with distinct computational tradeoffs

Converging evidence suggests the brain encodes time in dynamic patterns of neural activity, including neural sequences, ramping activity, and complex dynamics. Most temporal tasks, however, require more than just encoding time, and can have distinct computational requirements including the need to exhibit temporal scaling, generalize to novel contexts, or robustness to noise. It is not known how neural circuits can encode time and satisfy distinct computational requirements, nor is it known whether similar patterns of neural activity at the population level can exhibit dramatically different computational or generalization properties. To begin to answer these questions, we trained RNNs on two timing tasks based on behavioral studies. The tasks had different input structures but required producing identically timed output patterns. Using a novel framework we quantified whether RNNs encoded two intervals using either of three different timing strategies: scaling, absolute, or stimulus-specific dynamics. We found that similar neural dynamic patterns at the level of single intervals, could exhibit fundamentally different properties, including, generalization, the connectivity structure of the trained networks, and the contribution of excitatory and inhibitory neurons. Critically, depending on the task structure RNNs were better suited for generalization or robustness to noise. Further analysis revealed different connection patterns underlying the different regimes. Our results predict that apparently similar neural dynamic patterns at the population level (e.g., neural sequences) can exhibit fundamentally different computational properties in regards to their ability to generalize to novel stimuli and their robustness to noise—and that these differences are associated with differences in network connectivity and distinct contributions of excitatory and inhibitory neurons. We also predict that the task structure used in different experimental studies accounts for some of the experimentally observed variability in how networks encode time.     

人工神经网络模型及其在农业和生态学研究中的应用

对于一些复杂的农业生态系统,人们对其生态过程了解较少,且这些系统的不确定性和模糊性较大,用传统的方法难以模拟这些系统的行为,神经网络模型因为能较精确地模拟这些系统的行为,而引起生态学者们的广泛兴趣。该文着重介绍了误差逆传神经网络模型的结构、算法及其在农业和生态学中的应用研究。误差逆传神经网络模型一般采用三层神经网络模型结构,三层的神经网络模型能模拟任意复杂程度的连续函数,而且因为它的结构小而不容易产生与训练数据的过度吻合。误差逆传神经网络模型算法的主要特征是:利用当前的输入误差对权值进行调整。在生态学和农业研究中,误差逆传神经网络模型通常作为非线性函数模拟器用于预测作物产量、生物生产量、生物与环境之间的关系等。已有的研究表明:误差逆传神经网络模型的模拟精度要远远高于多元线性方程,类似于非线性方程,而在样本量足够的情况下,有一定的外推能力。但是误差逆传神经网络模型需要大量的样本量来保证所求取参数的可靠性,但这在实际研究中很难做到,因而限制了误差逆传神经网络模型的应用。近年来人们提出了强制训练停止、复合模型等多种技术来提高误差逆传神经网络模型的外推能力,也提出了Garson算法、敏感性分析以及随机化检验等技术对误差逆传神经网络模型的机理进行解释。误差逆传神经网络模型的真正优势在于模拟人们了解较少或不确定性和模糊性较大系统的行为,这些是传统模型所无法实现的,因而是对传统机理模型的重要补充。     

How robust are neural network models of stimulus generalization?

Artificial feed-forward neural networks are commonly used as a tool for modelling stimulus selection and animal signalling. A key finding of stimulus selection research has been generalization: if a given behaviour has been established to one stimulus, perceptually similar novel stimuli are likely to induce a similar response. Stimulus generalization, in feed-forward neural networks, automatically arises as a property of the network. This network property raises understandable concern regarding the sensitivity of the network to variation in its internal parameter values used in relation to its structure and to its training process. Researchers must have confidence that the predictions of their model follow from the underlying biology that they deliberately incorporated in the model, and not from often arbitrary choices about model implementation. We study how network training and parameter perturbations influence the qualitative and quantitative behaviour of a simple but general network. Specifically, for models of stimulus control we study the effect that parameter variation has on the shape of the generalization curves produced by the network. We show that certain network and training conditions produce undesirable artifacts that need to be avoided (or at least understood) when modelling stimulus selection.     

Inferring general relations between network characteristics from specific network ensembles

Different network models have been suggested for the topology underlying complex interactions in natural systems. These models are aimed at replicating specific statistical features encountered in real-world networks. However, it is rarely considered to which degree the results obtained for one particular network class can be extrapolated to real-world networks. We address this issue by comparing different classical and more recently developed network models with respect to their ability to generate networks with large structural variability. In particular, we consider the statistical constraints which the respective construction scheme imposes on the generated networks. After having identified the most variable networks, we address the issue of which constraints are common to all network classes and are thus suitable candidates for being generic statistical laws of complex networks. In fact, we find that generic, not model-related dependencies between different network characteristics do exist. This makes it possible to infer global features from local ones using regression models trained on networks with high generalization power. Our results confirm and extend previous findings regarding the synchronization properties of neural networks. Our method seems especially relevant for large networks, which are difficult to map completely, like the neural networks in the brain. The structure of such large networks cannot be fully sampled with the present technology. Our approach provides a method to estimate global properties of under-sampled networks in good approximation. Finally, we demonstrate on three different data sets (C. elegans neuronal network, R. prowazekii metabolic network, and a network of synonyms extracted from Roget's Thesaurus) that real-world networks have statistical relations compatible with those obtained using regression models.     

The ability of neuronal networks to generalize using an induction method

Inductive generalization means the ability of a neural network to learn a given algorithm using incomplete information about it. A consideration based on the information theory leads to a simple equation connecting characteristics of the network with those of the algorithm to be learned. The main conclusion is that the most efficient generalization is achieved on the networks with minimal complexity sufficient for realization of the algorithm under consideration. The obtained equation is compared with the results of computer simulations for a universal neural network obtained in the present paper as well as by other workers. A good agreement is observed between theoretical predictions for generalization efficacy and results of computer simulations.     

Deep inference of seabird dives from GPS-only records: Performance and generalization properties

At-sea behaviour of seabirds have received significant attention in ecology over the last decades as it is a key process in the ecology and fate of these populations. It is also, through the position of top predator that these species often occupy, a relevant and integrative indicator of the dynamics of the marine ecosystems they rely on. Seabird trajectories are recorded through the deployment of GPS, and a variety of statistical approaches have been tested to infer probable behaviours from these location data. Recently, deep learning tools have shown promising results for the segmentation and classification of animal behaviour from trajectory data. Yet, these approaches have not been widely used and investigation is still needed to identify optimal network architecture and to demonstrate their generalization properties. From a database of about 300 foraging trajectories derived from GPS data deployed simultaneously with pressure sensors for the identification of dives, this work has benchmarked deep neural network architectures trained in a supervised manner for the prediction of dives from trajectory data. It first confirms that deep learning allows better dive prediction than usual methods such as Hidden Markov Models. It also demonstrates the generalization properties of the trained networks for inferring dives distribution for seabirds from other colonies and ecosystems. In particular, convolutional networks trained on Peruvian boobies from a specific colony show great ability to predict dives of boobies from other colonies and from distinct ecosystems. We further investigate accross-species generalization using a transfer learning strategy known as ‘fine-tuning’. Starting from a convolutional network pre-trained on Guanay cormorant data reduced by two the size of the dataset needed to accurately predict dives in a tropical booby from Brazil. We believe that the networks trained in this study will provide relevant starting point for future fine-tuning works for seabird trajectory segmentation.     

Using artificially generated spectral data to improve protein secondary structure prediction from Fourier transform infrared spectra of proteins

Secondary structures of proteins have been predicted using neural networks from their Fourier transform infrared spectra. To improve the generalization ability of the neural networks, the training data set has been artificially increased by linear interpolation. The leave-one-out approach has been used to demonstrate the applicability of the method. Bayesian regularization has been used to train the neural networks and the predictions have been further improved by the maximum-likelihood estimation method. The networks have been tested and standard error of prediction (SEP) of 4.19% for alpha helix, 3.49% for beta sheet, and 3.15% for turns have been achieved. The results indicate that there is a significant decrease in the SEP for each type of structure parameter compared to previous works.     

Artificial neural networks and their use in quantitative pathology

A brief general introduction to artificial neural networks is presented, examining in detail the structure and operation of a prototype net developed for the solution of a simple pattern recognition problem in quantitative pathology. The process by which a neural network learns through example and gradually embodies its knowledge as a distributed representation is discussed, using this example. The application of neurocomputer technology to problems in quantitative pathology is explored, using real-world and illustrative examples. Included are examples of the use of artificial neural networks for pattern recognition, database analysis and machine vision. In the context of these examples, characteristics of neural nets, such as their ability to tolerate ambiguous, noisy and spurious data and spontaneously generalize from known examples to handle unfamiliar cases, are examined. Finally, the strengths and deficiencies of a connectionist approach are compared to those of traditional symbolic expert system methodology. It is concluded that artificial neural networks, used in conjunction with other nonalgorithmic artificial intelligence techniques and traditional algorithmic processing, may provide useful software engineering tools for the development of systems in quantitative pathology.     

Artificial neural network properties associated with wiring patterns in the visual projections of vertebrates and arthropods

We model the functioning of different wiring schemes in visual projections using artificial neural networks and so speculate on selective factors underlying taxonomic variation in neural architecture. We model the high connective overlap of vertebrates (where networks have a dense mesh of connections) and the less overlapping, more modular architecture of arthropods. We also consider natural variation in these basic wiring schemes. Generally, arthropod networks are as efficient or more efficient in functioning compared to vertebrate networks. They do not show the confusion effect (decreasing targeting accuracy with increasing input group size), and they train as well or better. Arthropod networks are, however, generally poorer at reconstructing novel inputs. The ability of vertebrate networks to effectively process novel stimuli could promote behavioral sophistication and drive the evolution of vertebrate wiring schemes. Vertebrate networks with less connective overlap have, surprisingly, similar or superior properties compared to those with high connective overlap. Thus, the partial connective overlap seen in real vertebrate visual projections may be an optimal, evolved solution. Arthropod networks with and without whole-cell neural connections within neural layers have similar properties. This indicates that neural connections mediated by offshoots of single cells (dendrites) may be fundamental to generating the confusion effect.     

Deep-learning-based 3D cellular force reconstruction directly from volumetric images

The forces exerted by single cells in the three-dimensional (3D) environments play a crucial role in modulating cellular functions and behaviors closely related to physiological and pathological processes. Cellular force microscopy (CFM) provides a feasible solution for quantifying mechanical interactions, which usually regains cellular forces from deformation information of extracellular matrices embedded with fluorescent beads. Owing to computational complexity, traditional 3D-CFM is usually extremely time consuming, which makes it challenging for efficient force recovery and large-scale sample analysis. With the aid of deep neural networks, this study puts forward a novel, data-driven 3D-CFM to reconstruct 3D cellular force fields directly from volumetric images with random fluorescence patterns. The deep-learning-based network is established through stacking deep convolutional neural networks (DCNN) and specific function layers. Some necessary physical information associated with constitutive relation of extracellular matrix material is coupled to the data-driven network. The mini-batch stochastic-gradient-descent and back-propagation algorithms are introduced to ensure its convergence and training efficiency. The networks not only have good generalization ability and robustness but also can recover 3D cellular forces directly from the input fluorescence image pairs. Particularly, the computational efficiency of the deep-learning-based network is at least one to two orders of magnitude higher than that of traditional 3D-CFM. This study provides a novel scheme for developing high-performance 3D-CFM to quantitatively characterize mechanical interactions between single cells and surrounding extracellular matrices, which is of vital importance for quantitative investigations in biomechanics and mechanobiology.     

Tuning diversity in bagged ensembles

In this paper, we investigate how the level of diversity amongst individual neural networks in a bagged ensemble can significantly influence overall ensemble generalization performance. We propose a new technique that tunes this diversity so that ensemble generalization performance is optimized and evaluate its performance on benchmark regression data-sets.     

A benchmark for how well neural nets generalize

This paper discusses the problem of extending the domain of learning sets and introduces HERBIE, a program which achieves this through graphical procedures rather than via neural networks. It is argued that for theoretical reasons HERBIE is well-suited to serving as a benchmark for measuring generalization efficacy, and therefore to serving as a means of testing claims of emergent distributed intelligence in neural nets. The successful results of tests of HERBIE as a pattern recognizer are presented, and HERBIE's behavior is favorably compared to neural nets for several real generalization problems. Finally, applications of HERBIE independent of its serving as a generalization benchmark, particularly in the area of cognitive science, are discussed.     

Generalization and the evolution of symmetry preferences

Biological displays are often symmetrical, and there is growing evidence that receivers are sensitive to these symmetries. One explanation for the evolution of such sensitivity is that symmetry reflects the quality of the signaller. An alternative is that the sensitivity may arise as a by-product of general properties of biological recognition systems. In line with the latter idea, simulations of the recognition process based on simple, artificial neural networks have suggested that generalization can give rise to preferences for particular symmetrical stimuli. However, it is not clear from these studies exactly how the preferences emerge, and to what extent the results are relevant to biological recognition systems. Here, we employ a different class of recognition models (gradient interaction models) to demonstrate more clearly how generalization can generate a preference for symmetrical variants of a display. We also point out that the predictions of the gradient interaction and network-based models regarding the effects of generalization closely match the results from empirical studies of stimulus control. Our analysis demonstrates that the effects of generalization cannot be ignored when studying the evolution of symmetry preferences and symmetric signals.     

A survey of partially connected neural networks

Almost all artificial neural networks are by default fully connected, which often implies a high redundancy and complexity. Little research has been devoted to the study of partially connected neural networks, despite its potential advantages like reduced training and recall time, improved generalization capabilities, reduced hardware requirements, as well as being a step closer to biological reality. This publication presents an extensive survey of the various kinds of partially connected neural networks, clustered into a clear framework, followed by a detailed comparative discussion.     

Multi-scale training of a large backpropagation net

The shortcomings of automatic learning in neural networks can be overcome by incorporating features of the problem into the weight space. We teach a multi-layer perceptron to recognize hand-printed characters by using a multi-scale method. This leads to better performance for learning rates and for generalization than direct use of back-propagation.     

人工神经网络模型在水稻群体分蘖动态模拟中外推能力的测试

研究利用人工神经网络模型 ,以水稻群体分蘖动态为例 ,采用交互验证和独立验证的方式 ,对水稻生长 BP网络模型进行了训练与模拟 ,其结果与水稻群体分蘖的积温统计模型、基本动力学模型和复合分蘖模型进行了比较。研究结果表明 ,神经网络模型具有一定的外推能力 ,但其外推能力依赖于大量的训练样本。神经网络模型具有较好的拟合能力 ,是因为有较多的模型参数 ,因此对神经网络模型的训练需要大量的参数来保证其参数不致过度吻合。具有外推能力神经网络模型的最少训练样本数应大于 6 .75倍于神经网络参数数目 ,小于 13.5倍于神经网络参数数目。因此在应用神经网络模型时 ,如果神经网络模型包括较多的输入变量时 ,可考虑采用主成分分析、对应分析等技术对输入变量进行信息综合 ,相应地减少网络模型的参数。另一方面 ,当训练样本不足时 ,最好只用神经网络模型对同一系统的情况进行模拟 ,应谨慎使用神经网络模型进行外推。神经网络模型给作物模拟研究的科学工作者提供了一个“傻瓜”式工具 ,对数学建模不熟悉的农业研究人员 ,人工神经网络可以替代数学建模进行仿真实验 ;对于精通数学建模的研究人员来说 ,它至少是一种补充和可作为比较的非线性数据处理方法     

Neural-space generalization of a topological transformation

An investigation is performed to assess the generalization capability found in neural network paradigms to approximate a 2-dimensional coordinate (topological) transformation. A developed strategy uses the example to give a physical meaning to what is meant by generalization. The example shows how to use a neural network paradigm to generalize a two-degree of freedom topological transformation from cartesian end-point coordinates to corresponding joint angle coordinates based only on examples of the mapping. The importance of this example is that it provides a clear understanding of how and what a neural network is actually communications and brings a theoretical idea to a useful understanding. When examples characterize the topology, a collective generalization property begins to emerge and the network learns the topology. If the network is presented with additional examples of the transformation, the network can generate the corresponding joint angles to any unseen position, that is, by generalization. It is also significant that the network's generalization property emerges from the network based on very few training examples! Further, the networks power exists with very few neurons. Results suggest the use of the paradigm's generalization capability to provide solutions to unknown or intractable algorithms for applications.     

Cortical mechanisms of sensory learning and object recognition

Learning about the world through our senses constrains our ability to recognise our surroundings. Experience shapes perception. What is the neural basis for object recognition and how are learning-induced changes in recognition manifested in neural populations? We consider first the location of neurons that appear to be critical for object recognition, before describing what is known about their function. Two complementary processes of object recognition are considered: discrimination among diagnostic object features and generalization across non-diagnostic features. Neural plasticity appears to underlie the development of discrimination and generalization for a given set of features, though tracking these changes directly over the course of learning has remained an elusive task.     

A hierarchical classification of first-order recurrent neural networks

We provide a decidable hierarchical classification of first-order recurrent neural networks made up of McCulloch and Pitts cells. This classification is achieved by proving an equivalence result between such neural networks and deterministic Büuchi automata, and then translating the Wadge classification theory from the abstract machine to the neural network context. The obtained hierarchy of neural networks is proved to have width 2 and height omega + 1, and a decidability procedure of this hierarchy is provided. Notably, this classification is shown to be intimately related to the attractive properties of the considered networks.     

Effect of the topology and delayed interactions in neuronal networks synchronization

As important as the intrinsic properties of an individual nervous cell stands the network of neurons in which it is embedded and by virtue of which it acquires great part of its responsiveness and functionality. In this study we have explored how the topological properties and conduction delays of several classes of neural networks affect the capacity of their constituent cells to establish well-defined temporal relations among firing of their action potentials. This ability of a population of neurons to produce and maintain a millisecond-precise coordinated firing (either evoked by external stimuli or internally generated) is central to neural codes exploiting precise spike timing for the representation and communication of information. Our results, based on extensive simulations of conductance-based type of neurons in an oscillatory regime, indicate that only certain topologies of networks allow for a coordinated firing at a local and long-range scale simultaneously. Besides network architecture, axonal conduction delays are also observed to be another important factor in the generation of coherent spiking. We report that such communication latencies not only set the phase difference between the oscillatory activity of remote neural populations but determine whether the interconnected cells can set in any coherent firing at all. In this context, we have also investigated how the balance between the network synchronizing effects and the dispersive drift caused by inhomogeneities in natural firing frequencies across neurons is resolved. Finally, we show that the observed roles of conduction delays and frequency dispersion are not particular to canonical networks but experimentally measured anatomical networks such as the macaque cortical network can display the same type of behavior.