Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, e.g. fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of RNNs may be useful both as a model of brain function and as a machine learning tool. We illustrate the use of the rRNN by an application to the online decoding (i.e. recognition) of human kinematics.     

Impacts of nested forward validation techniques on machine learning and regression waste disposal time series models

Dataset partitioning and validation techniques are required in all artificial neural network based waste models. However, there is currently no consensual approach on the validation techniques. This study examines the effects of three time series nested forward validation techniques (rolling origin - RO, rolling window - RW, and growing window - GW) on total municipal waste disposal estimates using recurrent neural network (RNN) models, and benchmarks model performance with respect to multiple linear regression (MLR) models. Validation selection techniques appear important to waste disposal time series model construction and evaluation. Sample size is found as an important factor on model accuracy for both RNN and MLR models. Better performance in Trial RW4 is observed, probably due to a more consistent testing set in 2019. Overall, the MAPE of the waste disposal models ranging from 10.4% to 12.7%. Both GW and RO validation techniques appear appropriate for RNN waste models. However, MLR waste models are more sensitive to the dataset characteristics, and RO validation technique appears more suitable to MLR models. It is found that data characteristics are more important than training period duration. It is recommended data set normality and skewness be examined for waste disposal modeling.     

Novel recurrent neural network for modelling biological networks: Oscillatory p53 interaction dynamics

Understanding the control of cellular networks consisting of gene and protein interactions and their emergent properties is a central activity of Systems Biology research. For this, continuous, discrete, hybrid, and stochastic methods have been proposed. Currently, the most common approach to modelling accurate temporal dynamics of networks is ordinary differential equations (ODE). However, critical limitations of ODE models are difficulty in kinetic parameter estimation and numerical solution of a large number of equations, making them more suited to smaller systems. In this article, we introduce a novel recurrent artificial neural network (RNN) that addresses above limitations and produces a continuous model that easily estimates parameters from data, can handle a large number of molecular interactions and quantifies temporal dynamics and emergent systems properties. This RNN is based on a system of ODEs representing molecular interactions in a signalling network. Each neuron represents concentration change of one molecule represented by an ODE. Weights of the RNN correspond to kinetic parameters in the system and can be adjusted incrementally during network training. The method is applied to the p53-Mdm2 oscillation system – a crucial component of the DNA damage response pathways activated by a damage signal. Simulation results indicate that the proposed RNN can successfully represent the behaviour of the p53-Mdm2 oscillation system and solve the parameter estimation problem with high accuracy. Furthermore, we presented a modified form of the RNN that estimates parameters and captures systems dynamics from sparse data collected over relatively large time steps. We also investigate the robustness of the p53-Mdm2 system using the trained RNN under various levels of parameter perturbation to gain a greater understanding of the control of the p53-Mdm2 system. Its outcomes on robustness are consistent with the current biological knowledge of this system. As more quantitative data become available on individual proteins, the RNN would be able to refine parameter estimation and mapping of temporal dynamics of individual signalling molecules as well as signalling networks as a system. Moreover, RNN can be used to modularise large signalling networks.     

Recurrent Neural Networks are universal approximators

Recurrent Neural Networks (RNN) have been developed for a better understanding and analysis of open dynamical systems. Still the question often arises if RNN are able to map every open dynamical system, which would be desirable for a broad spectrum of applications. In this article we give a proof for the universal approximation ability of RNN in state space model form and even extend it to Error Correction and Normalized Recurrent Neural Networks.     

White Blood Cells Image Classification Using Deep Learning with Canonical Correlation Analysis

White Blood Cells play an important role in observing the health condition of an individual. The opinion related to blood disease involves the identification and characterization of a patient's blood sample. Recent approaches employ Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and merging of CNN and RNN models to enrich the understanding of image content. From beginning to end, training of big data in medical image analysis has encouraged us to discover prominent features from sample images. A single cell patch extraction from blood sample techniques for blood cell classification has resulted in the good performance rate. However, these approaches are unable to address the issues of multiple cells overlap. To address this problem, the Canonical Correlation Analysis (CCA) method is used in this paper. CCA method views the effects of overlapping nuclei where multiple nuclei patches are extracted, learned and trained at a time. Due to overlapping of blood cell images, the classification time is reduced, the dimension of input images gets compressed and the network converges faster with more accurate weight parameters. Experimental results evaluated using publicly available database show that the proposed CNN and RNN merging model with canonical correlation analysis determines higher accuracy compared to other state-of-the-art blood cell classification techniques.     

Engineering aspects of biological ion channels--from biosensors to computational models for permeation

This paper presents two important engineering aspects of biological ion channels-how to build sensors out of gramicidin channels and how to construct computational models for ion channel permeation. We describe our recent research in these areas, potential challenges and possible solutions.     

Summary of the DREAM8 Parameter Estimation Challenge: Toward Parameter Identification for Whole-Cell Models

Whole-cell models that explicitly represent all cellular components at the molecular level have the potential to predict phenotype from genotype. However, even for simple bacteria, whole-cell models will contain thousands of parameters, many of which are poorly characterized or unknown. New algorithms are needed to estimate these parameters and enable researchers to build increasingly comprehensive models. We organized the Dialogue for Reverse Engineering Assessments and Methods (DREAM) 8 Whole-Cell Parameter Estimation Challenge to develop new parameter estimation algorithms for whole-cell models. We asked participants to identify a subset of parameters of a whole-cell model given the model’s structure and in silico “experimental” data. Here we describe the challenge, the best performing methods, and new insights into the identifiability of whole-cell models. We also describe several valuable lessons we learned toward improving future challenges. Going forward, we believe that collaborative efforts supported by inexpensive cloud computing have the potential to solve whole-cell model parameter estimation.     

深度学习在药物活性预测研究中的应用

药物从研发到临床应用需要耗费较长的时间，研发期间的投入成本可高达十几亿元。而随着医药研发与人工智能的结合以及生物信息学的飞速发展，药物活性相关数据急剧增加，传统的实验手段进行药物活性预测已经难以满足药物研发的需求。借助算法来辅助药物研发，解决药物研发中的各种问题能够大大推动药物研发进程。传统机器学习方法尤其是随机森林、支持向量机和人工神经网络在药物活性方面能够达到较高的预测精度。深度学习由于具有多层神经网络，模型可以接收高维的输入变量且不需要人工限定数据输入特征，可以拟合较为复杂的函数模型，应用于药物研发可以进一步提高各个环节的效率。在药物活性预测中应用较为广泛的深度学习模型主要是深度神经网络（deep neural networks，DNN）、循环神经网络（recurrent neural networks，RNN）和自编码器（auto encoder，AE），而生成对抗网络（generative adversarial networks，GAN）由于其生成数据的能力常常被用来和其他模型结合进行数据增强。近年来深度学习在药物分子活性预测方面的研究和应用综述表明，深度学习模型的准确度和效率均高于传统实验方法和传统机器学习方法。因此，深度学习模型有望成为药物研发领域未来十年最重要的辅助计算模型。     

Osteocytes but not osteoblasts directly build mineralized bone structures

Bone-forming osteoblasts have been a cornerstone of bone biology for more than a century. Most research toward bone biology and bone diseases center on osteoblasts. Overlooked are the 90% of bone cells, called osteocytes. This study aims to test the hypothesis that osteocytes but not osteoblasts directly build mineralized bone structures, and that defects in osteocytes lead to the onset of hypophosphatemia rickets. The hypothesis was tested by developing and modifying multiple imaging techniques, including both in vivo and in vitro models plus two types of hypophosphatemia rickets models (Dmp1-null and Hyp, Phex mutation mice), and Dmp1-Cre induced high level of β-catenin models. Our key findings were that osteocytes (not osteoblasts) build bone similar to the construction of a high-rise building, with a wire mesh frame (i.e., osteocyte dendrites) and cement (mineral matrices secreted from osteocytes), which is a lengthy and slow process whose mineralization direction is from the inside toward the outside. When osteoblasts fail to differentiate into osteocytes but remain highly active in Dmp-1-null or Hyp mice, aberrant and poor bone mineralization occurs, caused by a sharp increase in Wnt-β-catenin signaling. Further, the constitutive expression of β-catenin in osteocytes recaptures a similar osteomalacia phenotype as shown in Dmp1 null or Hyp mice. Thus, we conclude that osteocytes directly build bone, and osteoblasts with a short life span serve as a precursor to osteocytes, which challenges the existing dogma.     

Choosing where to look next in a mutation sequence space: Active Learning of informative p53 cancer rescue mutants

MOTIVATION: Many biomedical projects would benefit from reducing the time and expense of in vitro experimentation by using computer models for in silico predictions. These models may help determine which expensive biological data are most useful to acquire next. Active Learning techniques for choosing the most informative data enable biologists and computer scientists to optimize experimental data choices for rapid discovery of biological function. To explore design choices that affect this desirable behavior, five novel and five existing Active Learning techniques, together with three control methods, were tested on 57 previously unknown p53 cancer rescue mutants for their ability to build classifiers that predict protein function. The best of these techniques, Maximum Curiosity, improved the baseline accuracy of 56-77%. This article shows that Active Learning is a useful tool for biomedical research, and provides a case study of interest to others facing similar discovery challenges.     

Prediction of C alpha-H...O and C alpha-H...pi interactions in proteins using recurrent neural network

In this study, an attempt has been made to develop a method for predicting weak hydrogen bonding interactions, namely, C alpha-H...O and C alpha-H...pi interactions in proteins using artificial neural network. Both standard feed-forward neural network (FNN) and recurrent neural networks (RNN) have been trained and tested using five-fold cross-validation on a non-homologous dataset of 2298 protein chains where no pair of sequences has more than 25% sequence identity. It has been found that the prediction accuracy varies with the separation distance between donor and acceptor residues. The maximum sensitivity achieved with RNN for C alpha-H...O is 51.2% when donor and acceptor residues are four residues apart (i.e. at delta D-A = 4) and for C alpha-H...pi is 82.1% at delta D-A = 3. The performance of RNN is increased by 1-3% for both types of interactions when PSIPRED predicted protein secondary structure is used. Overall, RNN performs better than feed-forward networks at all separation distances between donor-acceptor pair for both types of interactions. Based on the observations, a web server CHpredict (available at http://www.imtech.res.in/raghava/chpredict/) has been developed for predicting donor and acceptor residues in C alpha-H...O and C alpha-H...pi interactions in proteins.     

Integrative Structural Biology in the Era of Accurate Structure Prediction

Characterizing the three-dimensional structure of macromolecules is central to understanding their function. Traditionally, structures of proteins and their complexes have been determined using experimental techniques such as X-ray crystallography, NMR, or cryo-electron microscopy—applied individually or in an integrative manner. Meanwhile, however, computational methods for protein structure prediction have been improving their accuracy, gradually, then suddenly, with the breakthrough advance by AlphaFold2, whose models of monomeric proteins are often as accurate as experimental structures. This breakthrough foreshadows a new era of computational methods that can build accurate models for most monomeric proteins. Here, we envision how such accurate modeling methods can combine with experimental structural biology techniques, enhancing integrative structural biology. We highlight the challenges that arise when considering multiple structural conformations, protein complexes, and polymorphic assemblies. These challenges will motivate further developments, both in modeling programs and in methods to solve experimental structures, towards better and quicker investigation of structure–function relationships.     

Genome Writing: Current Progress and Related Applications

The ultimate goal of synthetic biology is to build customized cells or organisms to meet specific industrial or medical needs. The most important part of the customized cell is a synthetic genome. Advanced genomic writing technologies are required to build such an artificial genome. Recently, the partially-completed synthetic yeast genome project represents a milestone in this field. In this mini review, we briefly introduce the techniques for de novo genome synthesis and genome editing. Furthermore, we summarize recent research progresses and highlight several applications in the synthetic genome field. Finally, we discuss current challenges and future prospects.     

The MYpop toolbox: Putting yeast stress responses in cellular context on single cell and population scales

Systems biology holds the promise to integrate multiple sources of information in order to build ever more complete models of cellular function. To do this, the field must overcome two significant challenges. First, the current strategy to model average cells must be replaced with population based models accounting for cell‐to‐cell variability. Second, models must be integrated with each other and with basic cellular function. This requires a core model of cellular physiology as well as a multiscale simulation platform to support large‐scale simulation of culture or tissues from single cells. Here, we present such a simulation platform with a core model of yeast physiology as scaffold to integrate and simulate SBML models. The software automates this integration helping users simulate their model of choice in context of the cell division cycle. We benchmark model merging, simulation and analysis by integrating a minimal model of osmotic stress into the core model and analyzing it. We characterize the effect of single cell differences on the dynamics of osmoadaptation, estimating when normal cell growth is resumed and obtaining an explanation for experimentally observed glycerol dynamics based on population dynamics. Hence, the platform can be used to reconcile single cell and population level data.     

The algorithmic anatomy of model-based evaluation

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.     

Amplification of the sequences displaying the pattern RNY in the RNA world: the translation --> translation/replication hypothesis

Based on previous considerations published in J. theor. Biol., new analyses of the organization of the genetic system are reported in this paper. We show that theoretical considerations about the order observed in the genetic code table support the idea of a primitive self-aminoacylation process achieved by primordial tRNAs. The physico-chemical constraints connected with this process may explain why a primitive genetic system predominantly uses sequences with the codonic pattern RNN (R=purine; Y=pyrimidine; N=any of the four bases) to polymerize the amino acids into peptides through translation. These considerations lead us to propose the Translation --> Translation/Replication hypothesis, which may explain why only RNA sequences with the pattern RNY, instead of less restrictive RNN, are susceptible to amplification. Using these ideas, supported by properties of symmetry, features of the genetic code may be connected with the replication of specific RNA sequences in the RNA world.     

Inference of genetic regulatory networks with recurrent neural network models using particle swarm optimization

Genetic regulatory network inference is critically important for revealing fundamental cellular processes, investigating gene functions, and understanding their relations. The availability of time series gene expression data makes it possible to investigate the gene activities of whole genomes, rather than those of only a pair of genes or among several genes. However, current computational methods do not sufficiently consider the temporal behavior of this type of data and lack the capability to capture the complex nonlinear system dynamics. We propose a recurrent neural network (RNN) and particle swarm optimization (PSO) approach to infer genetic regulatory networks from time series gene expression data. Under this framework, gene interaction is explained through a connection weight matrix. Based on the fact that the measured time points are limited and the assumption that the genetic networks are usually sparsely connected, we present a PSO-based search algorithm to unveil potential genetic network constructions that fit well with the time series data and explore possible gene interactions. Furthermore, PSO is used to train the RNN and determine the network parameters. Our approach has been applied to both synthetic and real data sets. The results demonstrate that the RNN/PSO can provide meaningful insights in understanding the nonlinear dynamics of the gene expression time series and revealing potential regulatory interactions between genes.     

Anti-Inflammatory Activity of Lauraceae Plant Species and Prediction Models Based on Their Metabolomics Profiling Data

The Lauraceae is a botanical family known for its anti-inflammatory potential. However, several species have not yet been studied. Thus, this work aimed to screen the anti-inflammatory activity of this plant family and to build statistical prediction models. The methodology was based on the statistical analysis of high-resolution liquid chromatography coupled with mass spectrometry data and the ex vivo anti-inflammatory activity of plant extracts. The ex vivo results demonstrated significant anti-inflammatory activity for several of these plants for the first time. The sample data were applied to build anti-inflammatory activity prediction models, including the partial least square acquired, artificial neural network, and stochastic gradient descent, which showed adequate fitting and predictive performance. Key anti-inflammatory markers, such as aporphine and benzylisoquinoline alkaloids were annotated with confidence level 2. Additionally, the validated prediction models proved to be useful for predicting active extracts using metabolomics data and studying their most bioactive metabolites.