Fides: Reliable trust-region optimization for parameter estimation of ordinary differential equation models

Ordinary differential equation (ODE) models are widely used to study biochemical reactions in cellular networks since they effectively describe the temporal evolution of these networks using mass action kinetics. The parameters of these models are rarely known a priori and must instead be estimated by calibration using experimental data. Optimization-based calibration of ODE models on is often challenging, even for low-dimensional problems. Multiple hypotheses have been advanced to explain why biochemical model calibration is challenging, including non-identifiability of model parameters, but there are few comprehensive studies that test these hypotheses, likely because tools for performing such studies are also lacking. Nonetheless, reliable model calibration is essential for uncertainty analysis, model comparison, and biological interpretation.We implemented an established trust-region method as a modular Python framework (fides) to enable systematic comparison of different approaches to ODE model calibration involving a variety of Hessian approximation schemes. We evaluated fides on a recently developed corpus of biologically realistic benchmark problems for which real experimental data are available. Unexpectedly, we observed high variability in optimizer performance among different implementations of the same mathematical instructions (algorithms). Analysis of possible sources of poor optimizer performance identified limitations in the widely used Gauss-Newton, BFGS and SR1 Hessian approximation schemes. We addressed these drawbacks with a novel hybrid Hessian approximation scheme that enhances optimizer performance and outperforms existing hybrid approaches. When applied to the corpus of test models, we found that fides was on average more reliable and efficient than existing methods using a variety of criteria. We expect fides to be broadly useful for ODE constrained optimization problems in biochemical models and to be a foundation for future methods development.     

Quantifying uncertainty,variability and likelihood for ordinary differential equation models

Background  

In many applications, ordinary differential equation (ODE) models are subject to uncertainty or variability in initial conditions and parameters. Both, uncertainty and variability can be quantified in terms of a probability density function on the state and parameter space.     

The effect of habitat fragmentation on cyclic population dynamics: a reduction to ordinary differential equations

Habitat fragmentation is known to be a key factor affecting population dynamics. In a previous study by Strohm and Tyson (Bull Math Biol 71:1323?C1348, 2009), the effect of habitat fragmentation on cyclic population dynamics was studied using spatially explicit predator?Cprey models with four different sets of reaction terms. The difficulty with spatially explicit models is that often analytical tractability is lost and the mechanisms behind the behaviour of the models are difficult to analyse. In this study, we employ a simplification procedure based on a Fourier series first-term truncation of the spatially explicit models Strohm and Tyson (Bull Math Biol 71:1323?C1348, 2009) to obtain spatially implicit models. These simpler models capture the main features of the spatially explicit models and can be used to explain the dynamics observed by Strohm and Tyson. We find that the spatially implicit models and the spatially explicit models produce similar responses to habitat fragmentation for larger high-quality patch sizes. Additionally, we find that the critical patch size of the spatially implicit models provides an upper bound on the critical patch size of the spatially explicit models. Finally, we derive an approximation of the multi-patch habitat by a single-patch habitat with partial flux boundary conditions which allows for a lower bound on the critical patch size to be calculated.     

Stability,sensitivity and uncertainty rates in the flow equations of ecological models

In previous work by the authors about dynamic system modeling, basic ecosystems concepts and their application to ecological modeling theory were formalized. Measuring how a variable effects certain processes leads to improvements in dynamic systems modelling and facilitates the author’s study of system diversity in which model sensitivity is a key theme. Initially, some variables and their numeric data are used for modeling. Predictions from the constructed models depend on these data. Generic study of sensitivity aims to show to what degree model behavior is altered by modification of some specific data. If small variations cause important modifications in the model’s global behavior then the model is very sensitive in relation to the variables used. If uncertain systems are considered, then it is important to submit the system to extreme situations and analyze its behavior. In this article, some indexes of uncertainty will be defined in order to determine the variables' influence in the case of extreme changes. This will permit analysis of the system’s sensitivity in relation to several simulations.     

Approximate reduction of linear population models governed by stochastic differential equations: application to multiregional models

In this work we develop approximate aggregation techniques in the context of slow-fast linear population models governed by stochastic differential equations and apply the results to the treatment of populations with spatial heterogeneity. Approximate aggregation techniques allow one to transform a complex system involving many coupled variables and in which there are processes with different time scales, by a simpler reduced model with a fewer number of ‘global’ variables, in such a way that the dynamics of the former can be approximated by that of the latter. In our model we contemplate a linear fast deterministic process together with a linear slow process in which the parameters are affected by additive noise, and give conditions for the solutions corresponding to positive initial conditions to remain positive for all times. By letting the fast process reach equilibrium we build a reduced system with a lesser number of variables, and provide results relating the asymptotic behaviour of the first- and second-order moments of the population vector for the original and the reduced system. The general technique is illustrated by analysing a multiregional stochastic system in which dispersal is deterministic and the rate growth of the populations in each patch is affected by additive noise.     

Identification in an anaerobic batch system: global sensitivity analysis, multi-start strategy and optimization criterion selection

Several mathematical models have been developed in anaerobic digestion systems and a variety of methods have been used for parameter estimation and model validation. However, structural and parametric identifiability questions are relatively seldom addressed in the reported AD modeling studies. This paper presents a 3-step procedure for the reliable estimation of a set of kinetic and stoichiometric parameters in a simplified model of the anaerobic digestion process. This procedure includes the application of global sensitivity analysis, which allows to evaluate the interaction among the identified parameters, multi-start strategy that gives a picture of the possible local minima and the selection of optimization criteria or cost functions. This procedure is applied to the experimental data collected from a lab-scale sequencing batch reactor. Two kinetic parameters and two stoichiometric coefficients are estimated and their accuracy was also determined. The classical least-squares cost function appears to be the best choice in this case study.     

A reconciliation of local and global models for bone remodeling through optimization theory

Remodeling rules with either a global or a local mathematical form have been proposed for load-bearing bones in the literature. In the local models, the bone architecture (shape, density) is related to the strains/energies sensed at any point in the bone, while in the global models, a criterion believed to be applicable to the whole bone is used. In the present paper, a local remodeling rule with a strain "error" form is derived as the necessary condition for the optimum of a global remodeling criterion, suggesting that many of the local error-driven remodeling rules may have corresponding global optimization-based criteria. The global criterion proposed in the present study is a trade-off between the cost of metabolic growth and use, mathematically represented by the mass, and the cost of failure, mathematically represented by the total strain energy. The proposed global criterion is shown to be related to the optimality criteria methods of structural optimization by the equivalence of the model solution and the fully stressed solution for statically determinate structures. In related work, the global criterion is applied to simulate the strength recovery in bones with screw holes left behind after removal of fracture fixation plates. The results predicted by the model are shown to be in good agreement with experimental results, leading to the conclusion that load-bearing bones are structures with optimal shape and property for their function.     

The Treeterbi and Parallel Treeterbi algorithms: efficient, optimal decoding for ordinary, generalized and pair HMMs

MOTIVATION: Hidden Markov models (HMMs) and generalized HMMs been successfully applied to many problems, but the standard Viterbi algorithm for computing the most probable interpretation of an input sequence (known as decoding) requires memory proportional to the length of the sequence, which can be prohibitive. Existing approaches to reducing memory usage either sacrifice optimality or trade increased running time for reduced memory. RESULTS: We developed two novel decoding algorithms, Treeterbi and Parallel Treeterbi, and implemented them in the TWINSCAN/N-SCAN gene-prediction system. The worst case asymptotic space and time are the same as for standard Viterbi, but in practice, Treeterbi optimally decodes arbitrarily long sequences with generalized HMMs in bounded memory without increasing running time. Parallel Treeterbi uses the same ideas to split optimal decoding across processors, dividing latency to completion by approximately the number of available processors with constant average overhead per processor. Using these algorithms, we were able to optimally decode all human chromosomes with N-SCAN, which increased its accuracy relative to heuristic solutions. We also implemented Treeterbi for Pairagon, our pair HMM based cDNA-to-genome aligner. AVAILABILITY: The TWINSCAN/N-SCAN/PAIRAGON open source software package is available from http://genes.cse.wustl.edu.     

Parameter optimization by using differential elimination: a general approach for introducing constraints into objective functions

Background

The investigation of network dynamics is a major issue in systems and synthetic biology. One of the essential steps in a dynamics investigation is the parameter estimation in the model that expresses biological phenomena. Indeed, various techniques for parameter optimization have been devised and implemented in both free and commercial software. While the computational time for parameter estimation has been greatly reduced, due to improvements in calculation algorithms and the advent of high performance computers, the accuracy of parameter estimation has not been addressed.

Results

We propose a new approach for parameter optimization by using differential elimination, to estimate kinetic parameter values with a high degree of accuracy. First, we utilize differential elimination, which is an algebraic approach for rewriting a system of differential equations into another equivalent system, to derive the constraints between kinetic parameters from differential equations. Second, we estimate the kinetic parameters introducing these constraints into an objective function, in addition to the error function of the square difference between the measured and estimated data, in the standard parameter optimization method. To evaluate the ability of our method, we performed a simulation study by using the objective function with and without the newly developed constraints: the parameters in two models of linear and non-linear equations, under the assumption that only one molecule in each model can be measured, were estimated by using a genetic algorithm (GA) and particle swarm optimization (PSO). As a result, the introduction of new constraints was dramatically effective: the GA and PSO with new constraints could successfully estimate the kinetic parameters in the simulated models, with a high degree of accuracy, while the conventional GA and PSO methods without them frequently failed.

Conclusions

The introduction of new constraints in an objective function by using differential elimination resulted in the drastic improvement of the estimation accuracy in parameter optimization methods. The performance of our approach was illustrated by simulations of the parameter optimization for two models of linear and non-linear equations, which included unmeasured molecules, by two types of optimization techniques. As a result, our method is a promising development in parameter optimization.

     

Dynamical behavior for population genetics models of differential and difference equations with nonmonotone fitnesses

This paper studies the dynamical behavior of classical 2-dimensional models of continuously and discretely reproducing diploid populations with two alleles at one locus. The phase variables are allele frequency and population density. The genotype fitnesses are not assumed to be monotonically decreasing functions of density. Hence the mean fitness curve is more complicated than in the monotonic case. If genotype fitnesses are only density dependent, results concerning equilibrium stability are obtained similar to those for the monotonic case, and periodic solutions are precluded in the differential equation model. An example with one-hump genotype fitnesses is presented and analyzed.Research supported by funds provided by the USDA-Forest Service, Southeastern Forest Experiment Station, Pioneering (Population Genetics of Forest Trees) Research Unit, Raleigh, North Carolina     

Evaluation of habitat suitability index models by global sensitivity and uncertainty analyses: a case study for submerged aquatic vegetation

Habitat suitability index (HSI) models are commonly used to predict habitat quality and species distributions and are used to develop biological surveys, assess reserve and management priorities, and anticipate possible change under different management or climate change scenarios. Important management decisions may be based on model results, often without a clear understanding of the level of uncertainty associated with model outputs. We present an integrated methodology to assess the propagation of uncertainty from both inputs and structure of the HSI models on model outputs (uncertainty analysis: UA) and relative importance of uncertain model inputs and their interactions on the model output uncertainty (global sensitivity analysis: GSA). We illustrate the GSA/UA framework using simulated hydrology input data from a hydrodynamic model representing sea level changes and HSI models for two species of submerged aquatic vegetation (SAV) in southwest Everglades National Park: Vallisneria americana (tape grass) and Halodule wrightii (shoal grass). We found considerable spatial variation in uncertainty for both species, but distributions of HSI scores still allowed discrimination of sites with good versus poor conditions. Ranking of input parameter sensitivities also varied spatially for both species, with high habitat quality sites showing higher sensitivity to different parameters than low‐quality sites. HSI models may be especially useful when species distribution data are unavailable, providing means of exploiting widely available environmental datasets to model past, current, and future habitat conditions. The GSA/UA approach provides a general method for better understanding HSI model dynamics, the spatial and temporal variation in uncertainties, and the parameters that contribute most to model uncertainty. Including an uncertainty and sensitivity analysis in modeling efforts as part of the decision‐making framework will result in better‐informed, more robust decisions.     

A comparative investigation of certain difference equations and related differential equations: Implications for model-building

Many mathematical models for physical and biological problems have been and will be built in the form of differential equations or systems of such equations. With the advent of digital computers one has been able to find (approximate) solutions for equations that used to be intractable. Many of the mathematical techniques used in this area amount to replacing the given differential equations by appropriate difference equations, so that extensive research has been done into how to choose appropriate difference equations whose solutions are “good” approximations to the solutions of the given differential equations. The present paper investigates a different, although related problem. For many physical and biological phenomena the “continuum” type of thinking, that is at the basis of any differential equation, is not natural to the phenomenon, but rather constitutes an approximation to a basically discrete situation: in much work of this type the “infinitesimal step lengths” handled in the reasoning which leads up to the differential equation, are not really thought of as infinitesimally small, but as finite; yet, in the last stage of such reasoning, where the differential equation rises from the differentials, these “infinitesimal” step lengths are allowed to go to zero: that is where the above-mentioned approximation comes in. Under this kind of circumstances, it seems more natural tobuild themodel as adiscrete difference equation (recurrence relation) from the start, without going through the painful, doubly approximative process of first, during the modeling stage, finding a differential equation to approximate a basically discrete situation, and then, for numerical computing purposes, approximating that differential equation by a difference scheme. The paper pursues this idea for some simple examples, where the old differential equation, though approximative in principle, had been at least qualitatively successful in describing certain phenomena, and shows that this idea, though plausible and sound in itself, does encounter some difficulties. The reason is that each differential equation, as it is set up in the way familiar to theoretical physicists and biologists, does correspond to a plethora of discrete difference equations, all of which in the limit (as step length→0) yield the same differential equation, but whose solutions, for not too small step length, are often widely different, some of them being quite irregular. The disturbing thing is that all these difference equations seem to adequately represent the same (physical or biological) reasoning as the differential equation in question. So, in order to choose the “right” difference equation, one may need to draw upon more detailed (physical or) biological considerations. All this does not say that one should not prefer discrete models for phenomena that seem to call for them; but only that their pursuit may require additional (physical or) biological refinement and insight. The paper also investigates some mathematical problems related to the fact of many difference equations being associated with one differential equation.     

Theory of metabolism regulation: a complete system of equations for regulation coefficients

Basic quantitative parameters of control in a metabolic system are considered: control coefficients of enzymes with respect to metabolic fluxes and concentrations, and in the case when there are conservation laws, the response coefficients of metabolic fluxes and concentrations to changes in the conserved sums of metabolite concentrations (e. g. conserved moieties). Relationships are obtained which generalize the well known connectivity relations for the case of metabolites binding by conservation laws. Additional relationships are obtained which complement the set of connectivity relations up to the complete system of equations for determining all the control coefficients. The control coefficients are expressed through the enzyme elasticity coefficients, steady state metabolic fluxes and concentrations. Formulas are derived which express response coefficients of flux and concentrations through the enzyme control and elasticity coefficients and metabolite concentrations.     

Graph-theoretic methods for the analysis of chemical and biochemical networks. I. Multistability and oscillations in ordinary differential equation models

A chemical mechanism is a model of a chemical reaction network consisting of a set of elementary reactions that express how molecules react with each other. In classical mass-action kinetics, a mechanism implies a set of ordinary differential equations (ODEs) which govern the time evolution of the concentrations. In this article, ODE models of chemical kinetics that have the potential for multiple positive equilibria or oscillations are studied. We begin by considering some methods of stability analysis based on the digraph of the Jacobian matrix. We then prove two theorems originally given by A. N. Ivanova which correlate the bifurcation structure of a mass-action model to the properties of a bipartite graph with nodes representing chemical species and reactions. We provide several examples of the application of these theorems.     

Microsatellite evolution: Markov transition functions for a suite of models

This paper takes from the collection of models considered by Whittaker et al. [2003. Likelihood-based estimation of microsatellite mutation rates. Genetics 164, 781-787] derived from direct observation of microsatellite mutation in parent-child pairs and provides analytical expressions for the probability distributions for the change in number of repeats over any given number of generations. The mathematical framework for this analysis is the theory of Markov processes. We find these expressions using two approaches, approximating by circulant matrices and solving a partial differential equation satisfied by the generating function. The impact of the differing choice of models is examined using likelihood estimates for time to most recent common ancestor. The analysis presented here may play a role in elucidating the connections between these two approaches and shows promise in reconciling differences between estimates for mutation rates based on Whittaker's approach and methods based on phylogenetic analyses.     

Integration of life cycle assessment,artificial neural networks,and metaheuristic optimization algorithms for optimization of tomato-based cropping systems in Iran

Purpose

The main purpose of this study was to evaluate the use of an integrated life cycle assessment (LCA), artificial neural network, and metaheuristic optimization model to improve the sustainability of tomato-based cropping systems in Iran. The model outputs the combination of input usage in a tomato cropping system, which leads to the highest economic output and the least environmental impact.

Methods

The LCA inventory was created using data from 114 open-field tomato farms in the Alborz Province of Iran during one growing period in 2015. Among all management components, the main focus was on irrigation management systems. The optimization problem was designed by integrating three indicators: carbon footprint (CF), benefit-cost ratio (BCR), and energy use efficiency (EUE) as the objective of field tomato production. The functional unit was 1 kg of tomato aligned with the system boundary of the cradle to market life cycle. Three artificial neural networks (ANNs) were applied to model relationships between the inputs and three indices (CF, BCR, and EUE) as the objective functions. Multi-objective genetic algorithm (MOGA) and multi-objective particle swarm optimization (MOPSO) were used to minimize the CF and maximize the BCR and EUE indicators. The abovementioned aims have been pursued by developing codes in MATLAB software.

Results and discussion

CF, BCR, and EUE were calculated to be 0.26 kg CO_2?eq (kg tomato)^?1, 1.8, and 0.5, respectively. MOGA results envisage the possibility of an increase of 86% and 50% in the EUE and BCR and a 43% reduction in the CF of tomato production systems. Moreover, EUE and BCR increased by 83% and 49%, and CF was reduced by 39% from the optimum results obtained from the MOPSO algorithm. It was revealed that in order to optimize field tomato production with the target objectives of this study, a large additional use for irrigation pipes, plastic, and machinery in comparison to current situation is required, while a large reduction of biocide, chemical fertilizer, and electricity consumption is indispensable.

Conclusions

According to the results of our study, it was concluded that the optimal solutions require a modernization of irrigation systems and a decrease in the consumption of chemical fertilizers and pesticides. The implementation of management options for such solutions is discussed.

     

Analysis of heterologous interacting systems by sedimentation velocity: curve fitting algorithms for estimation of sedimentation coefficients, equilibrium and kinetic constants

Analytical ultracentrifugation (AUC) has played and will continue to play an important role in the investigation of protein-protein, protein-DNA and protein-ligand interactions. A major advantage of AUC over other methods is that it allows the analysis of systems free in solution in nearly any buffer without worry about spurious interactions with a supporting matrix. Large amounts of high-quality data can be acquired in relatively short times. Advances in software for the treatment of AUC data over the last decade have eliminated many of the tedious aspects of AUC data analysis, allowing relatively rapid analysis of complicated systems that were previously unapproachable. A software package called sedanal is described that can perform global fits to AUC sedimentation velocity data obtained for both interacting and non-interacting, macromolecular multi-species, multi-component systems, by combining data from multiple runs over a range of sample concentrations and component ratios. Interaction parameters include both forward and reverse rate constants, or equilibrium constants, for each reaction, as well as concentration dependence of both sedimentation and diffusion coefficients. sedanal fits to time-difference data to eliminate time-independent systematic errors inherent in AUC data. The sedanal software package is based on the use of finite-element numerical solutions of the Lamm equation.