Internal representations for associative memory

We describe a class of feed forward neural network models for associative content addressable memory (ACAM) which utilize sparse internal representations for stored data. In addition to the input and output layers, our networks incorporate an intermediate processing layer which serves to label each stored memory and to perform error correction and association. We study two classes of internal label representations: the unary representation and various sparse, distributed representations. Finally, we consider storage of sparse data and sparsification of data. These models are found to have advantages in terms of storage capacity, hardware efficiency, and recall reliability when compared to the Hopfield model, and to possess analogies to both biological neural networks and standard digital computer memories.     

Incorporating Linguistic Knowledge for Learning Distributed Word Representations

Combined with neural language models, distributed word representations achieve significant advantages in computational linguistics and text mining. Most existing models estimate distributed word vectors from large-scale data in an unsupervised fashion, which, however, do not take rich linguistic knowledge into consideration. Linguistic knowledge can be represented as either link-based knowledge or preference-based knowledge, and we propose knowledge regularized word representation models (KRWR) to incorporate these prior knowledge for learning distributed word representations. Experiment results demonstrate that our estimated word representation achieves better performance in task of semantic relatedness ranking. This indicates that our methods can efficiently encode both prior knowledge from knowledge bases and statistical knowledge from large-scale text corpora into a unified word representation model, which will benefit many tasks in text mining.     

Space-Time Data fusion Under Error in Computer Model Output: An Application to Modeling Air Quality

Summary We provide methods that can be used to obtain more accurate environmental exposure assessment. In particular, we propose two modeling approaches to combine monitoring data at point level with numerical model output at grid cell level, yielding improved prediction of ambient exposure at point level. Extending our earlier downscaler model (Berrocal, V. J., Gelfand, A. E., and Holland, D. M. (2010b) . A spatio‐temporal downscaler for outputs from numerical models. Journal of Agricultural, Biological and Environmental Statistics 15, 176–197), these new models are intended to address two potential concerns with the model output. One recognizes that there may be useful information in the outputs for grid cells that are neighbors of the one in which the location lies. The second acknowledges potential spatial misalignment between a station and its putatively associated grid cell. The first model is a Gaussian Markov random field smoothed downscaler that relates monitoring station data and computer model output via the introduction of a latent Gaussian Markov random field linked to both sources of data. The second model is a smoothed downscaler with spatially varying random weights defined through a latent Gaussian process and an exponential kernel function, that yields, at each site, a new variable on which the monitoring station data is regressed with a spatial linear model. We applied both methods to daily ozone concentration data for the Eastern US during the summer months of June, July and August 2001, obtaining, respectively, a 5% and a 15% predictive gain in overall predictive mean square error over our earlier downscaler model ( Berrocal et al., 2010b ). Perhaps more importantly, the predictive gain is greater at hold‐out sites that are far from monitoring sites.     

Neural network input representations that produce accurate consensus sequences from DNA fragment assemblies.

MOTIVATION: Given inputs extracted from an aligned column of DNA bases and the underlying Perkin Elmer Applied Biosystems (ABI) fluorescent traces, our goal is to train a neural network to determine correctly the consensus base for the column. Choosing an appropriate network input representation is critical to success in this task. We empirically compare five representations; one uses only base calls and the others include trace information. RESULTS: We attained the most accurate results from networks that incorporate trace information into their input representations. Based on estimates derived from using 10-fold cross-validation, the best network topology produces consensus accuracies ranging from 99.26% to >99.98% for coverages from two to six aligned sequences. With a coverage of six, it makes only three errors in 20 000 consensus calls. In contrast, the network that only uses base calls in its input representation has over double that error rate: eight errors in 20 000 consensus calls. CONTACT: allex@cs.wisc.edu     

Mirrored STDP Implements Autoencoder Learning in a Network of Spiking Neurons

The autoencoder algorithm is a simple but powerful unsupervised method for training neural networks. Autoencoder networks can learn sparse distributed codes similar to those seen in cortical sensory areas such as visual area V1, but they can also be stacked to learn increasingly abstract representations. Several computational neuroscience models of sensory areas, including Olshausen & Field’s Sparse Coding algorithm, can be seen as autoencoder variants, and autoencoders have seen extensive use in the machine learning community. Despite their power and versatility, autoencoders have been difficult to implement in a biologically realistic fashion. The challenges include their need to calculate differences between two neuronal activities and their requirement for learning rules which lead to identical changes at feedforward and feedback connections. Here, we study a biologically realistic network of integrate-and-fire neurons with anatomical connectivity and synaptic plasticity that closely matches that observed in cortical sensory areas. Our choice of synaptic plasticity rules is inspired by recent experimental and theoretical results suggesting that learning at feedback connections may have a different form from learning at feedforward connections, and our results depend critically on this novel choice of plasticity rules. Specifically, we propose that plasticity rules at feedforward versus feedback connections are temporally opposed versions of spike-timing dependent plasticity (STDP), leading to a symmetric combined rule we call Mirrored STDP (mSTDP). We show that with mSTDP, our network follows a learning rule that approximately minimizes an autoencoder loss function. When trained with whitened natural image patches, the learned synaptic weights resemble the receptive fields seen in V1. Our results use realistic synaptic plasticity rules to show that the powerful autoencoder learning algorithm could be within the reach of real biological networks.     

Textrous!: Extracting Semantic Textual Meaning from Gene Sets

The un-biased and reproducible interpretation of high-content gene sets from large-scale genomic experiments is crucial to the understanding of biological themes, validation of experimental data, and the eventual development of plans for future experimentation. To derive biomedically-relevant information from simple gene lists, a mathematical association to scientific language and meaningful words or sentences is crucial. Unfortunately, existing software for deriving meaningful and easily-appreciable scientific textual ‘tokens’ from large gene sets either rely on controlled vocabularies (Medical Subject Headings, Gene Ontology, BioCarta) or employ Boolean text searching and co-occurrence models that are incapable of detecting indirect links in the literature. As an improvement to existing web-based informatic tools, we have developed Textrous!, a web-based framework for the extraction of biomedical semantic meaning from a given input gene set of arbitrary length. Textrous! employs natural language processing techniques, including latent semantic indexing (LSI), sentence splitting, word tokenization, parts-of-speech tagging, and noun-phrase chunking, to mine MEDLINE abstracts, PubMed Central articles, articles from the Online Mendelian Inheritance in Man (OMIM), and Mammalian Phenotype annotation obtained from Jackson Laboratories. Textrous! has the ability to generate meaningful output data with even very small input datasets, using two different text extraction methodologies (collective and individual) for the selecting, ranking, clustering, and visualization of English words obtained from the user data. Textrous!, therefore, is able to facilitate the output of quantitatively significant and easily appreciable semantic words and phrases linked to both individual gene and batch genomic data.     

The Fidelity of Dynamic Signaling by Noisy Biomolecular Networks

Cells live in changing, dynamic environments. To understand cellular decision-making, we must therefore understand how fluctuating inputs are processed by noisy biomolecular networks. Here we present a general methodology for analyzing the fidelity with which different statistics of a fluctuating input are represented, or encoded, in the output of a signaling system over time. We identify two orthogonal sources of error that corrupt perfect representation of the signal: dynamical error, which occurs when the network responds on average to other features of the input trajectory as well as to the signal of interest, and mechanistic error, which occurs because biochemical reactions comprising the signaling mechanism are stochastic. Trade-offs between these two errors can determine the system''s fidelity. By developing mathematical approaches to derive dynamics conditional on input trajectories we can show, for example, that increased biochemical noise (mechanistic error) can improve fidelity and that both negative and positive feedback degrade fidelity, for standard models of genetic autoregulation. For a group of cells, the fidelity of the collective output exceeds that of an individual cell and negative feedback then typically becomes beneficial. We can also predict the dynamic signal for which a given system has highest fidelity and, conversely, how to modify the network design to maximize fidelity for a given dynamic signal. Our approach is general, has applications to both systems and synthetic biology, and will help underpin studies of cellular behavior in natural, dynamic environments.     

Measuring internal representations from behavioral and brain data

The study of internal knowledge representations is a cornerstone of the research agenda in the interdisciplinary study of cognition. An influential proposal assumes that the brain uses its internal knowledge of the external world to constrain, in a top-down manner, high-dimensional sensory data into a lower-dimensional representation that enables perceptual decisions and other higher-level cognitive functions [1-9]. This proposal relies on a precise formulation of the observer-specific internal knowledge (i.e., the internal representations, or models) that guides reduction of the high-dimensional retinal input onto a low-dimensional code. Here, we directly revealed the content of subjective internal representations by instructing five observers to detect a face in the presence of only white noise, to force a pure top-down, knowledge-based task. We used reverse correlation methods to visualize each observer's internal representation that supports detection of an illusory face. Using reverse correlation again, this time applied to observers' electroencephalogram activity, we established where and when in the brain specific internal knowledge conceptually interprets the input white noise as a face. We show that internal representations can be reconstructed experimentally from behavioral and brain data, and that their content drives neural activity first over frontal and then over occipitotemporal cortex.     

Speech perception at the interface of neurobiology and linguistics

Speech perception consists of a set of computations that take continuously varying acoustic waveforms as input and generate discrete representations that make contact with the lexical representations stored in long-term memory as output. Because the perceptual objects that are recognized by the speech perception enter into subsequent linguistic computation, the format that is used for lexical representation and processing fundamentally constrains the speech perceptual processes. Consequently, theories of speech perception must, at some level, be tightly linked to theories of lexical representation. Minimally, speech perception must yield representations that smoothly and rapidly interface with stored lexical items. Adopting the perspective of Marr, we argue and provide neurobiological and psychophysical evidence for the following research programme. First, at the implementational level, speech perception is a multi-time resolution process, with perceptual analyses occurring concurrently on at least two time scales (approx. 20-80 ms, approx. 150-300 ms), commensurate with (sub)segmental and syllabic analyses, respectively. Second, at the algorithmic level, we suggest that perception proceeds on the basis of internal forward models, or uses an 'analysis-by-synthesis' approach. Third, at the computational level (in the sense of Marr), the theory of lexical representation that we adopt is principally informed by phonological research and assumes that words are represented in the mental lexicon in terms of sequences of discrete segments composed of distinctive features. One important goal of the research programme is to develop linking hypotheses between putative neurobiological primitives (e.g. temporal primitives) and those primitives derived from linguistic inquiry, to arrive ultimately at a biologically sensible and theoretically satisfying model of representation and computation in speech.     

The Deep Structure of Literary Representations

The cognitive rhetoricians have introduced the idea of cognitive domains into literary theory, but they have not yet developed a model for a comprehensive, species-typical structure of human motives. Evolutionary psychology can provide this model. Elemental human motives and basic emotions provide the deep structure of literary representations, and this deep structure serves to organize the particularities of circumstance and individual identity. Personal power and reproductive success are governing purposes in life and in literary representations. The concept of individual identity is necessary to literary representation, and a theory of literature based in evolutionary psychology has to incorporate models of personality. Literature and its oral antecedents organize experience in personally meaningful ways. They provide models of behavior and help regulate the complex cognitive machinery through which humans negotiate their social and cultural environments.     

Patient contrastive learning: A performant,expressive, and practical approach to electrocardiogram modeling

Supervised machine learning applications in health care are often limited due to a scarcity of labeled training data. To mitigate the effect of small sample size, we introduce a pre-training approach, Patient Contrastive Learning of Representations (PCLR), which creates latent representations of electrocardiograms (ECGs) from a large number of unlabeled examples using contrastive learning. The resulting representations are expressive, performant, and practical across a wide spectrum of clinical tasks. We develop PCLR using a large health care system with over 3.2 million 12-lead ECGs and demonstrate that training linear models on PCLR representations achieves a 51% performance increase, on average, over six training set sizes and four tasks (sex classification, age regression, and the detection of left ventricular hypertrophy and atrial fibrillation), relative to training neural network models from scratch. We also compared PCLR to three other ECG pre-training approaches (supervised pre-training, unsupervised pre-training with an autoencoder, and pre-training using a contrastive multi ECG-segment approach), and show significant performance benefits in three out of four tasks. We found an average performance benefit of 47% over the other models and an average of a 9% performance benefit compared to best model for each task. We release PCLR to enable others to extract ECG representations at https://github.com/broadinstitute/ml4h/tree/master/model_zoo/PCLR.     

Sentence alignment using feed forward neural network

Parallel corpora have become an essential resource for work in multi lingual natural language processing. However, sentence aligned parallel corpora are more efficient than non-aligned parallel corpora for cross language information retrieval and machine translation applications. In this paper, we present a new approach to align sentences in bilingual parallel corpora based on feed forward neural network classifier. A feature parameter vector is extracted from the text pair under consideration. This vector contains text features such as length, punctuate score, and cognate score values. A set of manually prepared training data has been assigned to train the feed forward neural network. Another set of data was used for testing. Using this new approach, we could achieve an error reduction of 60% over length based approach when applied on English-Arabic parallel documents. Moreover this new approach is valid for any language pair and it is quite flexible approach since the feature parameter vector may contain more/less or different features than that we used in our system such as lexical match feature.     

A stochastic collocation method for uncertainty quantification and propagation in cardiovascular simulations

Simulations of blood flow in both healthy and diseased vascular models can be used to compute a range of hemodynamic parameters including velocities, time varying wall shear stress, pressure drops, and energy losses. The confidence in the data output from cardiovascular simulations depends directly on our level of certainty in simulation input parameters. In this work, we develop a general set of tools to evaluate the sensitivity of output parameters to input uncertainties in cardiovascular simulations. Uncertainties can arise from boundary conditions, geometrical parameters, or clinical data. These uncertainties result in a range of possible outputs which are quantified using probability density functions (PDFs). The objective is to systemically model the input uncertainties and quantify the confidence in the output of hemodynamic simulations. Input uncertainties are quantified and mapped to the stochastic space using the stochastic collocation technique. We develop an adaptive collocation algorithm for Gauss-Lobatto-Chebyshev grid points that significantly reduces computational cost. This analysis is performed on two idealized problems--an abdominal aortic aneurysm and a carotid artery bifurcation, and one patient specific problem--a Fontan procedure for congenital heart defects. In each case, relevant hemodynamic features are extracted and their uncertainty is quantified. Uncertainty quantification of the hemodynamic simulations is done using (a) stochastic space representations, (b) PDFs, and (c) the confidence intervals for a specified level of confidence in each problem.     

Latent Semantics of Action Verbs Reflect Phonetic Parameters of Intensity and Emotional Content

Conjuring up our thoughts, language reflects statistical patterns of word co-occurrences which in turn come to describe how we perceive the world. Whether counting how frequently nouns and verbs combine in Google search queries, or extracting eigenvectors from term document matrices made up of Wikipedia lines and Shakespeare plots, the resulting latent semantics capture not only the associative links which form concepts, but also spatial dimensions embedded within the surface structure of language. As both the shape and movements of objects have been found to be associated with phonetic contrasts already in toddlers, this study explores whether articulatory and acoustic parameters may likewise differentiate the latent semantics of action verbs. Selecting 3 × 20 emotion-, face-, and hand-related verbs known to activate premotor areas in the brain, their mutual cosine similarities were computed using latent semantic analysis LSA, and the resulting adjacency matrices were compared based on two different large scale text corpora: HAWIK and TASA. Applying hierarchical clustering to identify common structures across the two text corpora, the verbs largely divide into combined mouth and hand movements versus emotional expressions. Transforming the verbs into their constituent phonemes, and projecting them into an articulatory space framed by tongue height and formant frequencies, the clustered small and large size movements appear differentiated by front versus back vowels corresponding to increasing levels of arousal. Whereas the clustered emotional verbs seem characterized by sequences of close versus open jaw produced phonemes, generating up- or downwards shifts in formant frequencies that may influence their perceived valence. Suggesting, that the latent semantics of action verbs reflect parameters of intensity and emotional polarity that appear correlated with the articulatory contrasts and acoustic characteristics of phonemes.     

Chronically deafferented sensory cortex recovers a grossly typical organization after allogenic hand transplantation

Amputation induces substantial reorganization of the body part somatotopy in primary sensory cortex (S1 complex, hereafter S1) [1, 2], and these effects of deafferentiation increase with time [3]. Determining whether these changes are reversible is critical for understanding the potential to recover from deafferenting injuries. Earlier BOLD fMRI data demonstrate increased S1 activity in response to stimulation of an allogenically transplanted hand [4]. Here, we report the first evidence that the representation of a transplanted hand can actually recapture the pre-amputation S1 hand territory. A 54-year-old male received a unilateral hand transplant 35 years after traumatic amputation of his right hand. Despite limited sensation, palmar tactile stimulation delivered 4 months post-transplant evoked contralateral S1 responses that were indistinguishable in location and amplitude from those detected in healthy matched controls. We find no evidence for persistent intrusion of representations of the face within the representation of the transplanted hand, although such intrusions are commonly reported in amputees [5, 6]. Our results suggest that even decades after complete deafferentiation, restoring afferent input to S1 leads to re-establishment of the gross hand representation within its original territory. Unexpectedly, large ipsilateral S1 responses accompanied sensory stimulation of the patient's intact hand. These may reflect a change in interhemispheric inhibition that could contribute to maintaining latent hand representations during the period of amputation.     

Predicting drug polypharmacology from cell morphology readouts using variational autoencoder latent space arithmetic

A variational autoencoder (VAE) is a machine learning algorithm, useful for generating a compressed and interpretable latent space. These representations have been generated from various biomedical data types and can be used to produce realistic-looking simulated data. However, standard vanilla VAEs suffer from entangled and uninformative latent spaces, which can be mitigated using other types of VAEs such as β-VAE and MMD-VAE. In this project, we evaluated the ability of VAEs to learn cell morphology characteristics derived from cell images. We trained and evaluated these three VAE variants—Vanilla VAE, β-VAE, and MMD-VAE—on cell morphology readouts and explored the generative capacity of each model to predict compound polypharmacology (the interactions of a drug with more than one target) using an approach called latent space arithmetic (LSA). To test the generalizability of the strategy, we also trained these VAEs using gene expression data of the same compound perturbations and found that gene expression provides complementary information. We found that the β-VAE and MMD-VAE disentangle morphology signals and reveal a more interpretable latent space. We reliably simulated morphology and gene expression readouts from certain compounds thereby predicting cell states perturbed with compounds of known polypharmacology. Inferring cell state for specific drug mechanisms could aid researchers in developing and identifying targeted therapeutics and categorizing off-target effects in the future.