Reinforcement Learning on Slow Features of High-Dimensional Input Streams

Humans and animals are able to learn complex behaviors based on a massive stream of sensory information from different modalities. Early animal studies have identified learning mechanisms that are based on reward and punishment such that animals tend to avoid actions that lead to punishment whereas rewarded actions are reinforced. However, most algorithms for reward-based learning are only applicable if the dimensionality of the state-space is sufficiently small or its structure is sufficiently simple. Therefore, the question arises how the problem of learning on high-dimensional data is solved in the brain. In this article, we propose a biologically plausible generic two-stage learning system that can directly be applied to raw high-dimensional input streams. The system is composed of a hierarchical slow feature analysis (SFA) network for preprocessing and a simple neural network on top that is trained based on rewards. We demonstrate by computer simulations that this generic architecture is able to learn quite demanding reinforcement learning tasks on high-dimensional visual input streams in a time that is comparable to the time needed when an explicit highly informative low-dimensional state-space representation is given instead of the high-dimensional visual input. The learning speed of the proposed architecture in a task similar to the Morris water maze task is comparable to that found in experimental studies with rats. This study thus supports the hypothesis that slowness learning is one important unsupervised learning principle utilized in the brain to form efficient state representations for behavioral learning.     

Investigating the efficacy of nonlinear dimensionality reduction schemes in classifying gene and protein expression studies

The recent explosion in procurement and availability of high-dimensional gene- and protein-expression profile datasets for cancer diagnostics has necessitated the development of sophisticated machine learning tools with which to analyze them. A major limitation in the ability to accurate classify these high-dimensional datasets stems from the 'curse of dimensionality', occurring in situations where the number of genes or peptides significantly exceeds the total number of patient samples. Previous attempts at dealing with this issue have mostly centered on the use of a dimensionality reduction (DR) scheme, Principal Component Analysis (PCA), to obtain a low-dimensional projection of the high-dimensional data. However, linear PCA and other linear DR methods, which rely on Euclidean distances to estimate object similarity, do not account for the inherent underlying nonlinear structure associated with most biomedical data. The motivation behind this work is to identify the appropriate DR methods for analysis of high-dimensional gene- and protein-expression studies. Towards this end, we empirically and rigorously compare three nonlinear (Isomap, Locally Linear Embedding, Laplacian Eigenmaps) and three linear DR schemes (PCA, Linear Discriminant Analysis, Multidimensional Scaling) with the intent of determining a reduced subspace representation in which the individual object classes are more easily discriminable.     

A hybrid neural network of addressable and content-addressable memory

We investigate the memory structure and retrieval of the brain and propose a hybrid neural network of addressable and content-addressable memory which is a special database model and can memorize and retrieve any piece of information (a binary pattern) both addressably and content-addressably. The architecture of this hybrid neural network is hierarchical and takes the form of a tree of slabs which consist of binary neurons with the same array. Simplex memory neural networks are considered as the slabs of basic memory units, being distributed on the terminal vertexes of the tree. It is shown by theoretical analysis that the hybrid neural network is able to be constructed with Hebbian and competitive learning rules, and some other important characteristics of its learning and memory behavior are also consistent with those of the brain. Moreover, we demonstrate the hybrid neural network on a set of ten binary numeral patters     

Dimensionality reduction for visualization of normal and pathological speech data

For an adequate analysis of pathological speech signals, a sizeable number of parameters is required, such as those related to jitter, shimmer and noise content. Often this kind of high-dimensional signal representation is difficult to understand, even for expert voice therapists and physicians. Data visualization of a high-dimensional dataset can provide a useful first step in its exploratory data analysis, facilitating an understanding about its underlying structure. In the present paper, eight dimensionality reduction techniques, both classical and recent, are compared on speech data containing normal and pathological speech. A qualitative analysis of their dimensionality reduction capabilities is presented. The transformed data are also quantitatively evaluated, using classifiers, and it is found that it may be advantageous to perform the classification process on the transformed data, rather than on the original. These qualitative and quantitative analyses allow us to conclude that a nonlinear, supervised method, called kernel local Fisher discriminant analysis is superior for dimensionality reduction in the actual context.     

Protein sequence classification using feature hashing

Recent advances in next-generation sequencing technologies have resulted in an exponential increase in the rate at which protein sequence data are being acquired. The k-gram feature representation, commonly used for protein sequence classification, usually results in prohibitively high dimensional input spaces, for large values of k. Applying data mining algorithms to these input spaces may be intractable due to the large number of dimensions. Hence, using dimensionality reduction techniques can be crucial for the performance and the complexity of the learning algorithms. In this paper, we study the applicability of feature hashing to protein sequence classification, where the original high-dimensional space is "reduced" by hashing the features into a low-dimensional space, using a hash function, i.e., by mapping features into hash keys, where multiple features can be mapped (at random) to the same hash key, and "aggregating" their counts. We compare feature hashing with the "bag of k-grams" approach. Our results show that feature hashing is an effective approach to reducing dimensionality on protein sequence classification tasks.     

A SOM projection technique with the growing structure for visualizing high-dimensional data

The Self-Organizing Map (SOM) is an efficient tool for visualizing high-dimensional data. In this paper, an intuitive and effective SOM projection method is proposed for mapping high-dimensional data onto the two-dimensional grid structure with a growing self-organizing mechanism. In the learning phase, a growing SOM is trained and the growing cell structure is used as the baseline framework. In the ordination phase, the new projection method is used to map the input vector so that the input data is mapped to the structure of the SOM without having to plot the weight values, resulting in easy visualization of the data. The projection method is demonstrated on four different data sets, including a 118 patent data set and a 399 checical abstract data set related to polymer cements, with promising results and a significantly reduced network size.     

Effects of pre-exposure to an experimental chamber on memory trace retrieval in aggressive and submissive mice during extinction of conditioning

Mice with a submissive stereotype of behaviour developed a fast habituation to novelty of environment. A subsequent training revealed a deficit of passive avoidance learning and prolongation of the memory trace extinction. Aggressive mice are characterised by a delay of the habituation and absence of any significant changes in the memory trace retrieval. The findings suggest that responses to environmental novelty and use of its estimation in learning and retention of memory traces are essentially predetermined by the basic strategy of behaviour.     

Embedding Responses in Spontaneous Neural Activity Shaped through Sequential Learning

Recent experimental measurements have demonstrated that spontaneous neural activity in the absence of explicit external stimuli has remarkable spatiotemporal structure. This spontaneous activity has also been shown to play a key role in the response to external stimuli. To better understand this role, we proposed a viewpoint, “memories-as-bifurcations,” that differs from the traditional “memories-as-attractors” viewpoint. Memory recall from the memories-as-bifurcations viewpoint occurs when the spontaneous neural activity is changed to an appropriate output activity upon application of an input, known as a bifurcation in dynamical systems theory, wherein the input modifies the flow structure of the neural dynamics. Learning, then, is a process that helps create neural dynamical systems such that a target output pattern is generated as an attractor upon a given input. Based on this novel viewpoint, we introduce in this paper an associative memory model with a sequential learning process. Using a simple Hebbian-type learning, the model is able to memorize a large number of input/output mappings. The neural dynamics shaped through the learning exhibit different bifurcations to make the requested targets stable upon an increase in the input, and the neural activity in the absence of input shows chaotic dynamics with occasional approaches to the memorized target patterns. These results suggest that these dynamics facilitate the bifurcations to each target attractor upon application of the corresponding input, which thus increases the capacity for learning. This theoretical finding about the behavior of the spontaneous neural activity is consistent with recent experimental observations in which the neural activity without stimuli wanders among patterns evoked by previously applied signals. In addition, the neural networks shaped by learning properly reflect the correlations of input and target-output patterns in a similar manner to those designed in our previous study.     

Memory reconsolidation: sensitivity of spatial memory to inhibition of protein synthesis in dorsal hippocampus during encoding and retrieval

Reconsolidation is a putative neuronal process in which the retrieval of a previously consolidated memory returns it to a labile state that is once again subject to stabilization. This study explored the idea that reconsolidation occurs in spatial memory when animals retrieve memory under circumstances in which new memory encoding is likely to occur. Control studies confirmed that intrahippocampal infusions of anisomycin inhibited protein synthesis locally and that the spatial training protocols we used are subject to overnight protein synthesis-dependent consolidation. We then compared the impact of anisomycin in two conditions: when memory retrieval occurred in a reference memory task after performance had reached asymptote over several days; and after a comparable extent of training of a delayed matching-to-place task in which new memory encoding was required each day. Sensitivity to intrahippocampal anisomycin was observed only in the protocol involving new memory encoding at the time of retrieval.     

Blood pressure variations real-time reflect the conditioned fear learning and memory

The conditioned fear learning and memory occurs when a neutral conditioned stimulus (CS) is paired with an aversive unconditioned stimulus (US). This process is critically dependent on the amygdala and inevitably involves blood pressure (BP) alterations. We hypothesized that BP variations could instantaneously reveal individual steps during conditioned fear learning and memory. An implanted telemetric probe was used to monitor the BP real-time in rats during training and testing sessions of the fear-potentiated startle. Our results showed that (i) the conditioned fear learning during the training sessions was reflected by light (CS)-induced rapid BP elevations and by electric shock (US)-evoked sympathetic tone elevations; (ii) these two BP-related parameters were not only negatively correlated with each other but also coupled to each other in the training session trials; (iii) both parameters closely predicted the performance of fear-potentiated startle on the next day; and (iv) although local blocking of one of the two fear-conditioned pathways in the training session partially inhibited fear learning, the fear memory retrieval still used both pathways. Altogether, real-time blood pressure variations faithfully revealed the critical steps involved in conditioned fear learning and memory, and our results supported a coupling between the cued learning and the post-shock calmness.     

Learning shapes spontaneous activity itinerating over memorized states

Learning is a process that helps create neural dynamical systems so that an appropriate output pattern is generated for a given input. Often, such a memory is considered to be included in one of the attractors in neural dynamical systems, depending on the initial neural state specified by an input. Neither neural activities observed in the absence of inputs nor changes caused in the neural activity when an input is provided were studied extensively in the past. However, recent experimental studies have reported existence of structured spontaneous neural activity and its changes when an input is provided. With this background, we propose that memory recall occurs when the spontaneous neural activity changes to an appropriate output activity upon the application of an input, and this phenomenon is known as bifurcation in the dynamical systems theory. We introduce a reinforcement-learning-based layered neural network model with two synaptic time scales; in this network, I/O relations are successively memorized when the difference between the time scales is appropriate. After the learning process is complete, the neural dynamics are shaped so that it changes appropriately with each input. As the number of memorized patterns is increased, the generated spontaneous neural activity after learning shows itineration over the previously learned output patterns. This theoretical finding also shows remarkable agreement with recent experimental reports, where spontaneous neural activity in the visual cortex without stimuli itinerate over evoked patterns by previously applied signals. Our results suggest that itinerant spontaneous activity can be a natural outcome of successive learning of several patterns, and it facilitates bifurcation of the network when an input is provided.     

Heat shock during the development of brain structures of Drosophila: the memory development in the l(1)ts403 mutant of Drosophila melanogaster

The structures and functions of many genes are homologous in Drosophila and humans. Therefore, studying pathological processes in Drosophila, in particular neurogenerative processes accompanied by progressive memory loss, helps to understand the ethiology of corresponding human disorders and to develop therapeutic strategies. It is believed that the development of neurogenerative diseases might result from alterations in the functioning of the heat shock/chaperone machinery. In view of this, we used Drosophila mutant l(1)ts403 with defective synthesis of heat shock proteins for studying learning and memory in a test of conditioned courtship suppression following a heat shock given at different developmental stages. High learning indices were registered immediately and 30 min after training both in the intact controls and in flies subjected to different developmental heat shocks. This indicated normal learning and memory acquisition in the mutant. At the same time, memory retention (3 h after training) suffered to different extent depending on the developmental stage. The remote effects of heat shock given during the formation of the mushroom bodies indicated the important role of this brain structure in the memory formation. The observed memory defects may result from alterations both in mRNA transport and in the functions of molecular chaperones in the l(1)ts403 mutant.     

Estimation of effective connectivity using multi-layer perceptron artificial neural network

Studies on interactions between brain regions estimate effective connectivity, (usually) based on the causality inferences made on the basis of temporal precedence. In this study, the causal relationship is modeled by a multi-layer perceptron feed-forward artificial neural network, because of the ANN’s ability to generate appropriate input–output mapping and to learn from training examples without the need of detailed knowledge of the underlying system. At any time instant, the past samples of data are placed in the network input, and the subsequent values are predicted at its output. To estimate the strength of interactions, the measure of “Causality coefficient” is defined based on the network structure, the connecting weights and the parameters of hidden layer activation function. Simulation analysis demonstrates that the method, called “CREANN” (Causal Relationship Estimation by Artificial Neural Network), can estimate time-invariant and time-varying effective connectivity in terms of MVAR coefficients. The method shows robustness with respect to noise level of data. Furthermore, the estimations are not significantly influenced by the model order (considered time-lag), and the different initial conditions (initial random weights and parameters of the network). CREANN is also applied to EEG data collected during a memory recognition task. The results implicate that it can show changes in the information flow between brain regions, involving in the episodic memory retrieval process. These convincing results emphasize that CREANN can be used as an appropriate method to estimate the causal relationship among brain signals.     

Forgetting, reminding, and remembering: the retrieval of lost spatial memory

Retrograde amnesia can occur after brain damage because this disrupts sites of storage, interrupts memory consolidation, or interferes with memory retrieval. While the retrieval failure account has been considered in several animal studies, recent work has focused mainly on memory consolidation, and the neural mechanisms responsible for reactivating memory from stored traces remain poorly understood. We now describe a new retrieval phenomenon in which rats' memory for a spatial location in a watermaze was first weakened by partial lesions of the hippocampus to a level at which it could not be detected. The animals were then reminded by the provision of incomplete and potentially misleading information—an escape platform in a novel location. Paradoxically, both incorrect and correct place information reactivated dormant memory traces equally, such that the previously trained spatial memory was now expressed. It was also established that the reminding procedure could not itself generate new learning in either the original environment, or in a new training situation. The key finding is the development of a protocol that definitively distinguishes reminding from new place learning and thereby reveals that a failure of memory during watermaze testing can arise, at least in part, from a disruption of memory retrieval.     

Effect of dimethoate and fenvalerate on conditioned avoidance and passive avoidance responses

Dimethoate at 24.75 and 49.5 mg/kg (i.e., 1/10 and 1/5th LD50 respectively) impaired the learning process and retrieval of memory in rats while it did not affect permanent memory. Fenvalerate / 10.12 and 20 mg/kg (i.e., 1/20 and 1/10th LD50 respectively) had no effect on learning process, retrieval of memory and permanent memory traces.     

深度学习在药物活性预测研究中的应用

药物从研发到临床应用需要耗费较长的时间，研发期间的投入成本可高达十几亿元。而随着医药研发与人工智能的结合以及生物信息学的飞速发展，药物活性相关数据急剧增加，传统的实验手段进行药物活性预测已经难以满足药物研发的需求。借助算法来辅助药物研发，解决药物研发中的各种问题能够大大推动药物研发进程。传统机器学习方法尤其是随机森林、支持向量机和人工神经网络在药物活性方面能够达到较高的预测精度。深度学习由于具有多层神经网络，模型可以接收高维的输入变量且不需要人工限定数据输入特征，可以拟合较为复杂的函数模型，应用于药物研发可以进一步提高各个环节的效率。在药物活性预测中应用较为广泛的深度学习模型主要是深度神经网络（deep neural networks，DNN）、循环神经网络（recurrent neural networks，RNN）和自编码器（auto encoder，AE），而生成对抗网络（generative adversarial networks，GAN）由于其生成数据的能力常常被用来和其他模型结合进行数据增强。近年来深度学习在药物分子活性预测方面的研究和应用综述表明，深度学习模型的准确度和效率均高于传统实验方法和传统机器学习方法。因此，深度学习模型有望成为药物研发领域未来十年最重要的辅助计算模型。     

Heat Shock during the Development of Central Structures of the Drosophila Brain: Memory Formation in the l(1)ts403 Mutant of Drosophila melanogaster

The structures and functions of many genes are homologous in Drosophilaand humans. Therefore, studying pathological processes in Drosophila, in particular neurogenerative processes accompanied by progressive memory loss, helps to understand the ethiology of corresponding human disorders and to develop therapeutic strategies. It is believed that the development of neurogenerative diseases might result from alterations in the functioning of the heat shock/chaperone machinery. In view of this, we used Drosophila mutant l(1)ts403 with defective synthesis of heat shock proteins for studying learning and memory in a test of conditioned courtship suppression following a heat shock given at different developmental stages. High learning indices were registered immediately and 30 min after training both in the intact controls and in flies subjected to different developmental heat shocks. This indicated normal learning and memory acquisition in the mutant. At the same time, memory retention (3 h after training) suffered to different extent depending on the developmental stage. The remote effects of heat shock given during the formation of the mushroom bodies indicated the important role of this brain structure in the memory formation. The observed memory defects may result from alterations both in mRNA transport and in the functions of molecular chaperones in the l(1)ts403 mutant.     

Estimating the dimensionality of the manifold underlying multi-electrode neural recordings

It is generally accepted that the number of neurons in a given brain area far exceeds the number of neurons needed to carry any specific function controlled by that area. For example, motor areas of the human brain contain tens of millions of neurons that control the activation of tens or at most hundreds of muscles. This massive redundancy implies the covariation of many neurons, which constrains the population activity to a low-dimensional manifold within the space of all possible patterns of neural activity. To gain a conceptual understanding of the complexity of the neural activity within a manifold, it is useful to estimate its dimensionality, which quantifies the number of degrees of freedom required to describe the observed population activity without significant information loss. While there are many algorithms for dimensionality estimation, we do not know which are well suited for analyzing neural activity. The objective of this study was to evaluate the efficacy of several representative algorithms for estimating the dimensionality of linearly and nonlinearly embedded data. We generated synthetic neural recordings with known intrinsic dimensionality and used them to test the algorithms’ accuracy and robustness. We emulated some of the important challenges associated with experimental data by adding noise, altering the nature of the embedding of the low-dimensional manifold within the high-dimensional recordings, varying the dimensionality of the manifold, and limiting the amount of available data. We demonstrated that linear algorithms overestimate the dimensionality of nonlinear, noise-free data. In cases of high noise, most algorithms overestimated the dimensionality. We thus developed a denoising algorithm based on deep learning, the “Joint Autoencoder”, which significantly improved subsequent dimensionality estimation. Critically, we found that all algorithms failed when the intrinsic dimensionality was high (above 20) or when the amount of data used for estimation was low. Based on the challenges we observed, we formulated a pipeline for estimating the dimensionality of experimental neural data.     

A model of associative learning in the mushroom body

The mushroom body is a prominent invertebrate neuropil strongly associated with learning and memory. We built a high-level computational model of this structure using simplified but realistic models of neurons and synapses, and developed a learning rule based on activity dependent pre-synaptic facilitation. We show that our model, which is consistent with mushroom body Drosophila data and incorporates Aplysia learning, is able to both acquire and later recall CS-US associations. We demonstrate that a highly divergent input connectivity to the mushroom body and strong periodic inhibition both serve to improve overall learning performance. We also examine the problem of how synaptic conductance, driven by successive training events, obtains a value appropriate for the stimulus being learnt. We employ two feedback mechanisms: one stabilises strength at an initial level appropriate for an association; another prevents strength increase for established associations.     

Kernel hierarchical gene clustering from microarray expression data

MOTIVATION: Unsupervised analysis of microarray gene expression data attempts to find biologically significant patterns within a given collection of expression measurements. For example, hierarchical clustering can be applied to expression profiles of genes across multiple experiments, identifying groups of genes that share similar expression profiles. Previous work using the support vector machine supervised learning algorithm with microarray data suggests that higher-order features, such as pairwise and tertiary correlations across multiple experiments, may provide significant benefit in learning to recognize classes of co-expressed genes. RESULTS: We describe a generalization of the hierarchical clustering algorithm that efficiently incorporates these higher-order features by using a kernel function to map the data into a high-dimensional feature space. We then evaluate the utility of the kernel hierarchical clustering algorithm using both internal and external validation. The experiments demonstrate that the kernel representation itself is insufficient to provide improved clustering performance. We conclude that mapping gene expression data into a high-dimensional feature space is only a good idea when combined with a learning algorithm, such as the support vector machine that does not suffer from the curse of dimensionality. AVAILABILITY: Supplementary data at www.cs.columbia.edu/compbio/hiclust. Software source code available by request.