Performance of likelihood ratio tests of evolutionary hypotheses under inadequate substitution models.

In recent years, likelihood ratio tests (LRTs) based on DNA and protein sequence data have been proposed for testing various evolutionary hypotheses. Because conducting an LRT requires an evolutionary model of nucleotide or amino acid substitution, which is almost always unknown, it becomes important to investigate the robustness of LRTs to violations of assumptions of these evolutionary models. Computer simulation was used to examine performance of LRTs of the molecular clock, transition/transversion bias, and among-site rate variation under different substitution models. The results showed that when correct models are used, LRTs perform quite well even when the DNA sequences are as short as 300 nt. However, LRTs were found to be biased under incorrect models. The extent of bias varies considerably, depending on the hypotheses tested, the substitution models assumed, and the lengths of the sequences used, among other things. A preliminary simulation study also suggests that LRTs based on parametric bootstrapping may be more sensitive to substitution models than are standard LRTs. When an assumed substitution model is grossly wrong and a more realistic model is available, LRTs can often reject the wrong model; thus, the performance of LRTs may be improved by using a more appropriate model. On the other hand, many factors of molecular evolution have not been considered in any substitution models so far built, and the possibility of an influence of this negligence on LRTs is often overlooked. The dependence of LRTs on substitution models calls for caution in interpreting test results and highlights the importance of clarifying the substitution patterns of genes and proteins and building more realistic models.     

Nonparametric comparison of two survival-time distributions in the presence of dependent censoring

When testing the null hypothesis that treatment arm-specific survival-time distributions are equal, the log-rank test is asymptotically valid when the distribution of time to censoring is conditionally independent of randomized treatment group given survival time. We introduce a test of the null hypothesis for use when the distribution of time to censoring depends on treatment group and survival time. This test does not make any assumptions regarding independence of censoring time and survival time. Asymptotic validity of this test only requires a consistent estimate of the conditional probability that the survival event is observed given both treatment group and that the survival event occurred before the time of analysis. However, by not making unverifiable assumptions about the data-generating mechanism, there exists a set of possible values of corresponding sample-mean estimates of these probabilities that are consistent with the observed data. Over this subset of the unit square, the proposed test can be calculated and a rejection region identified. A decision on the null that considers uncertainty because of censoring that may depend on treatment group and survival time can then be directly made. We also present a generalized log-rank test that enables us to provide conditions under which the ordinary log-rank test is asymptotically valid. This generalized test can also be used for testing the null hypothesis when the distribution of censoring depends on treatment group and survival time. However, use of this test requires semiparametric modeling assumptions. A simulation study and an example using a recent AIDS clinical trial are provided.     

Group sequential methods for an ordinal logistic random-effects model under misspecification

Interim analyses in clinical trials are planned for ethical as well as economic reasons. General results have been published in the literature that allow the use of standard group sequential methodology if one uses an efficient test statistic, e.g., when Wald-type statistics are used in random-effects models for ordinal longitudinal data. These models often assume that the random effects are normally distributed. However, this is not always the case. We will show that, when the random-effects distribution is misspecified in ordinal regression models, the joint distribution of the test statistics over the different interim analyses is still a multivariate normal distribution, but a sandwich-type correction to the covariance matrix is needed in order to obtain the correct covariance matrix. The independent increment structure is also investigated. A bias in estimation will occur due to the misspecification. However, we will also show that the treatment effect estimate will be unbiased under the null hypothesis, thus maintaining the type I error. Extensive simulations based on a toenail dermatophyte onychomycosis trial are used to illustrate our results.     

Testing random effects in linear mixed‐effects models with serially correlated errors

In linear mixed‐effects models, random effects are used to capture the heterogeneity and variability between individuals due to unmeasured covariates or unknown biological differences. Testing for the need of random effects is a nonstandard problem because it requires testing on the boundary of parameter space where the asymptotic chi‐squared distribution of the classical tests such as likelihood ratio and score tests is incorrect. In the literature several tests have been proposed to overcome this difficulty, however all of these tests rely on the restrictive assumption of i.i.d. measurement errors. The presence of correlated errors, which often happens in practice, makes testing random effects much more difficult. In this paper, we propose a permutation test for random effects in the presence of serially correlated errors. The proposed test not only avoids issues with the boundary of parameter space, but also can be used for testing multiple random effects and any subset of them. Our permutation procedure includes the permutation procedure in Drikvandi, Verbeke, Khodadadi, and Partovi Nia (2013) as a special case when errors are i.i.d., though the test statistics are different. We use simulations and a real data analysis to evaluate the performance of the proposed permutation test. We have found that random slopes for linear and quadratic time effects may not be significant when measurement errors are serially correlated.     

Using Regression Models to Analyze Randomized Trials: Asymptotically Valid Hypothesis Tests Despite Incorrectly Specified Models

Summary .  Regression models are often used to test for cause-effect relationships from data collected in randomized trials or experiments. This practice has deservedly come under heavy scrutiny, because commonly used models such as linear and logistic regression will often not capture the actual relationships between variables, and incorrectly specified models potentially lead to incorrect conclusions. In this article, we focus on hypothesis tests of whether the treatment given in a randomized trial has any effect on the mean of the primary outcome, within strata of baseline variables such as age, sex, and health status. Our primary concern is ensuring that such hypothesis tests have correct type I error for large samples. Our main result is that for a surprisingly large class of commonly used regression models, standard regression-based hypothesis tests (but using robust variance estimators) are guaranteed to have correct type I error for large samples, even when the models are incorrectly specified. To the best of our knowledge, this robustness of such model-based hypothesis tests to incorrectly specified models was previously unknown for Poisson regression models and for other commonly used models we consider. Our results have practical implications for understanding the reliability of commonly used, model-based tests for analyzing randomized trials.     

Selection of evolutionary models for phylogenetic hypothesis testing using parametric methods

Recent molecular studies have incorporated the parametric bootstrap method to test a priori hypotheses when the results of molecular based phylogenies are in conflict with these hypotheses. The parametric bootstrap requires the specification of a particular substitutional model, the parameters of which will be used to generate simulated, replicate DNA sequence data sets. It has been both suggested that, (a) the method appears robust to changes in the model of evolution, and alternatively that, (b) as realistic model of DNA substitution as possible should be used to avoid false rejection of a null hypothesis. Here we empirically evaluate the effect of suboptimal substitution models when testing hypotheses of monophyly with the parametric bootstrap using data sets of mtDNA cytochrome oxidase I and II (COI and COII) sequences for Macaronesian Calathus beetles, and mitochondrial 16S rDNA and nuclear ITS2 sequences for European Timarcha beetles. Whether a particular hypothesis of monophyly is rejected or accepted appears to be highly dependent on whether the nucleotide substitution model being used is optimal. It appears that a parameter rich model is either equally or less likely to reject a hypothesis of monophyly where the optimal model is unknown. A comparison of the performance of the Kishino–Hasegawa (KH) test shows it is not as severely affected by the use of suboptimal models, and overall it appears to be a less conservative method with a higher rate of failure to reject null hypotheses.     

Masting,mixtures and modes: are two models better than one?

A key hypothesis in population ecology is that synchronous and intermittent seed production, known as mast seeding, is driven by the alternating allocation of carbohydrates and mineral nutrients between growth and reproduction in different years, i.e. ‘resource switching’. Such behaviour may ultimately generate bimodal distributions of long‐term flower and seed production, and evidence of these patterns has been taken to support the resource switching hypothesis. Here, we show how a widely‐used statistical test of bimodality applied by many studies in different ecological contexts may fail to reject the null hypothesis that focal probability distributions are unimodal. Using data from five tussock grass species in South Island, New Zealand, we find clear evidence of bimodality only when flowering patterns are analyzed with probabilistic mixture models. Mixture models provide a theory oriented framework for testing hypotheses of mast seeding patterns, enabling the different responses underlying medium‐ and high‐ versus non‐ and low‐flowering years to be modelled more realistically by associating these with distinct probability distributions. Coupling theoretical expectations with more rigorous statistical approaches will empower ecologists to reject null hypotheses more often.     

Variance Calculations for Direct Adjusted Survival Curves,with Applications to Testing for No Treatment Effect

Several investigators have recently constructed survival curves adjusted for imbalances in prognostic factors by a method which we call direct adjustment. We present methods for calculating variances of these direct adjusted survival curves and their differences. Estimates of the adjusted curves, their variances, and the variances of their differences are compared for non-parametric (Kaplan-Meier), semi-parametric (Cox) and parametric (Weibull) models applied to censored exponential data. Semi-parametric proportional hazards models were nearly fully efficient for estimating differences in adjusted curves, but parametric estimates of individual adjusted curves may be substantially more precise. Standardized differences between direct adjusted survival curves may be used to test the null hypothesis of no treatment effect. This procedure may prove especially useful when the proportional hazards assumption is questionable.     

Ecologists should not use statistical significance tests to interpret simulation model results

Simulation models are widely used to represent the dynamics of ecological systems. A common question with such models is how changes to a parameter value or functional form in the model alter the results. Some authors have chosen to answer that question using frequentist statistical hypothesis tests (e.g. ANOVA). This is inappropriate for two reasons. First, p‐values are determined by statistical power (i.e. replication), which can be arbitrarily high in a simulation context, producing minuscule p‐values regardless of the effect size. Second, the null hypothesis of no difference between treatments (e.g. parameter values) is known a priori to be false, invalidating the premise of the test. Use of p‐values is troublesome (rather than simply irrelevant) because small p‐values lend a false sense of importance to observed differences. We argue that modelers should abandon this practice and focus on evaluating the magnitude of differences between simulations. Synthesis Researchers analyzing field or lab data often test ecological hypotheses using frequentist statistics (t‐tests, ANOVA, etc.) that focus on p‐values. Field and lab data usually have limited sample sizes, and p‐values are valuable for quantifying the probability of making incorrect inferences in that situation. However, modern ecologists increasingly rely on simulation models to address complex questions, and those who were trained in frequentist statistics often apply the hypothesis‐testing approach inappropriately to their simulation results. Our paper explains why p‐values are not informative for interpreting simulation models, and suggests better ways to evaluate the ecological significance of model results.     

THE BETA-BINOMIAL MODEL: ACCOUNTING FOR INTER-TRIAL VARIATION IN REPLICATED DIFFERENCE AND PREFERENCE TESTS

Binomial tests are commonly used in sensory difference and preference testing under the assumptions that choices are independent and choice probabilities do not vary from trial to trial. This paper addresses violations of the latter assumption (often referred to as overdispersion) and accounts for variation in inter-trial choice probabilities following the Beta distribution. Such variation could arise as a result of differences in test substrate from trial to trial, differences in sensory acuity among subjects or the existence of latent preference segments. In fact, it is likely that overdispersion occurs ubiquitously in product testing. Using the Binomial model for data in which there is inter-trial variation may lead to seriously misleading conclusions from a sensory difference or preference test. A simulation study in this paper based on product testing experience showed that when using a Binomial model for overdispersed Binomial data, Type I error may be 0.44 for a Binomial test specification corresponding to a level of 0.05. Underestimation of Type I error using the Binomial model may seriously undermine legal claims of product superiority in situations where overdispersion occurs. The Beta-Binomial (BB) model, an extension of the Binomial distribution, was developed to fit overdispersed Binomial data. Procedures for estimating and testing the parameters as well as testing for goodness of fit are discussed. Procedures for determining sample size and for calculating estimate precision and test power based on the BB model are given. Numerical examples and simulation results are also given in the paper. The BB model should improve the validity of sensory difference and preference testing.     

Toward ecologically explicit null models of nestedness

A community is "nested" when species assemblages in less rich sites form nonrandom subsets of those at richer sites. Conventional null models used to test for statistically nonrandom nestedness are under- or over-restrictive because they do not sufficiently isolate ecological processes of interest, which hinders ecological inference. We propose a class of null models that are ecologically explicit and interpretable. Expected values of species richness and incidence, rather than observed values, are used to create random presence-absence matrices for hypothesis testing. In our examples, based on six datasets, expected values were derived either by using an individually based random placement model or by fitting empirical models to richness data as a function of environmental covariates. We describe an algorithm for constructing unbiased null matrices, which permitted valid testing of our null models. Our approach avoids the problem of building too much structure into the null model, and enabled us to explicitly test whether observed communities were more nested than would be expected for a system structured solely by species-abundance and species-area or similar relationships. We argue that this test or similar tests are better determinants of whether a system is truly nested; a nested system should contain unique pattern not already predicted by more fundamental ecological principles such as species-area relationships. Most species assemblages we studied were not nested under these null models. Our results suggest that nestedness, beyond that which is explained by passive sampling processes, may not be as widespread as currently believed. These findings may help to improve the utility of nestedness as an ecological concept and conservation tool.     

A unified evaluation of differential vaccine efficacy

Many infectious diseases are well prevented by proper vaccination. However, when a vaccine is not completely efficacious, there is great interest in determining how the vaccine effect differs in subgroups conditional on measured immune responses postvaccination and also according to the type of infecting agent (eg, strain of a virus). The former is often called correlate of protection (CoP) analysis, while the latter has been called sieve analysis. We propose a unified framework for simultaneously assessing CoP and sieve effects of a vaccine in a large Phase III randomized trial. We use flexible parametric models treating times to infection from different agents as competing risks and estimated maximum likelihood to fit the models. The parametric models under competing risks allow for estimation of both cumulative incidence-based contrasts and instantaneous rates. We outline the assumptions with which we can link the observable data to the causal contrasts of interest, propose hypothesis testing procedures, and evaluate our proposed methods in an extensive simulation study.     

Testing superiority and non-inferiority hypotheses in active controlled clinical trials

Switching between testing for superiority and non-inferiority has been an important statistical issue in the design and analysis of active controlled clinical trial. In practice, it is often conducted with a two-stage testing procedure. It has been assumed that there is no type I error rate adjustment required when either switching to test for non-inferiority once the data fail to support the superiority claim or switching to test for superiority once the null hypothesis of non-inferiority is rejected with a pre-specified non-inferiority margin in a generalized historical control approach. However, when using a cross-trial comparison approach for non-inferiority testing, controlling the type I error rate sometimes becomes an issue with the conventional two-stage procedure. We propose to adopt a single-stage simultaneous testing concept as proposed by Ng (2003) to test both non-inferiority and superiority hypotheses simultaneously. The proposed procedure is based on Fieller's confidence interval procedure as proposed by Hauschke et al. (1999).     

Testing Pairwise Association between Spatially Autocorrelated Variables: A New Approach Using Surrogate Lattice Data

Background

Independence between observations is a standard prerequisite of traditional statistical tests of association. This condition is, however, violated when autocorrelation is present within the data. In the case of variables that are regularly sampled in space (i.e. lattice data or images), such as those provided by remote-sensing or geographical databases, this problem is particularly acute. Because analytic derivation of the null probability distribution of the test statistic (e.g. Pearson''s r) is not always possible when autocorrelation is present, we propose instead the use of a Monte Carlo simulation with surrogate data.

Methodology/Principal Findings

The null hypothesis that two observed mapped variables are the result of independent pattern generating processes is tested here by generating sets of random image data while preserving the autocorrelation function of the original images. Surrogates are generated by matching the dual-tree complex wavelet spectra (and hence the autocorrelation functions) of white noise images with the spectra of the original images. The generated images can then be used to build the probability distribution function of any statistic of association under the null hypothesis. We demonstrate the validity of a statistical test of association based on these surrogates with both actual and synthetic data and compare it with a corrected parametric test and three existing methods that generate surrogates (randomization, random rotations and shifts, and iterative amplitude adjusted Fourier transform). Type I error control was excellent, even with strong and long-range autocorrelation, which is not the case for alternative methods.

Conclusions/Significance

The wavelet-based surrogates are particularly appropriate in cases where autocorrelation appears at all scales or is direction-dependent (anisotropy). We explore the potential of the method for association tests involving a lattice of binary data and discuss its potential for validation of species distribution models. An implementation of the method in Java for the generation of wavelet-based surrogates is available online as supporting material.     

Measuring disease-specific quality of life in clinical trials.

While measurement of quality of life is a vital part of assessing the effect of treatment in many clinical trials, a measure that is responsive to clinically important change is often unavailable. Investigators are therefore faced with the challenge of constructing an index for a specific condition or even for a single trial. There are several stages in the development and testing of a quality-of-life measure: selecting an initial item pool, choosing the "best" items from that pool, deciding on questionnaire format, pretesting the instrument, and demonstrating the responsiveness and validity of the instrument. At each stage the investigator must choose between a rigorous, time-consuming approach to questionnaire construction that will establish the clinical relevance, responsiveness and validity of the instrument and a more efficient, less costly strategy that leaves reproducibility, responsiveness and validity untested. This article describes these options and outlines a pragmatic approach that yields consistently satisfactory disease-specific measures of quality of life.     

Selecting factors predictive of heterogeneity in multivariate event time data

In multivariate survival analysis, investigators are often interested in testing for heterogeneity among clusters, both overall and within specific classes. We represent different hypotheses about the heterogeneity structure using a sequence of gamma frailty models, ranging from a null model with no random effects to a full model having random effects for each class. Following a Bayesian approach, we define prior distributions for the frailty variances consisting of mixtures of point masses at zero and inverse-gamma densities. Since frailties with zero variance effectively drop out of the model, this prior allocates probability to each model in the sequence, including the overall null hypothesis of homogeneity. Using a counting process formulation, the conditional posterior distributions of the frailties and proportional hazards regression coefficients have simple forms. Posterior computation proceeds via a data augmentation Gibbs sampling algorithm, a single run of which can be used to obtain model-averaged estimates of the population parameters and posterior model probabilities for testing hypotheses about the heterogeneity structure. The methods are illustrated using data from a lung cancer trial.     

Resampling-based multiple testing methods with covariate adjustment: application to investigation of antiretroviral drug susceptibility

Summary .   Identifying genetic mutations that cause clinical resistance to antiretroviral drugs requires adjustment for potential confounders, such as the number of active drugs in a HIV-infected patient's regimen other than the one of interest. Motivated by this problem, we investigated resampling-based methods to test equal mean response across multiple groups defined by HIV genotype, after adjustment for covariates. We consider construction of test statistics and their null distributions under two types of model: parametric and semiparametric. The covariate function is explicitly specified in the parametric but not in the semiparametric approach. The parametric approach is more precise when models are correctly specified, but suffer from bias when they are not; the semiparametric approach is more robust to model misspecification, but may be less efficient. To help preserve type I error while also improving power in both approaches, we propose resampling approaches based on matching of observations with similar covariate values. Matching reduces the impact of model misspecification as well as imprecision in estimation. These methods are evaluated via simulation studies and applied to a data set that combines results from a variety of clinical studies of salvage regimens. Our focus is on relating HIV genotype to viral susceptibility to abacavir after adjustment for the number of active antiretroviral drugs (excluding abacavir) in the patient's regimen.     

The role of parametric linkage methods in complex trait analyses using microsatellites

Many investigators of complexly inherited familial traits bypass classical segregation analysis to perform model-free genome-wide linkage scans. Because model-based or parametric linkage analysis may be the most powerful means to localize genes when a model can be approximated, model-free statistics may result in a loss of power to detect linkage. We performed limited segregation analyses on the electrophysiological measurements that have been collected for the Collaborative Study on the Genetics of Alcoholism. The resulting models are used in whole-genome scans. Four genomic regions provided a model-based LOD > 2 and only 3 of these were detected (p < 0.05) by a model-free approach. We conclude that parametric methods, using even over-simplified models of complex phenotypes, may complement nonparametric methods and decrease false positives.     

Estimating a treatment effect with repeated measurements accounting for varying effectiveness duration

To assess treatment efficacy in clinical trials, certain clinicaloutcomes are repeatedly measured over time for the same subject.The difference in their means may characterize a treatment effect.Since treatment effectiveness lag and saturation times may exist,erosion of treatment effect often occurs during the observationperiod. Instead of using models based on ad hoc parametric orpurely nonparametric time-varying coefficients, we model thetreatment effectiveness durations, which are the time intervalsbetween the lag and saturation times. Then we use some meanresponse models to include such treatment effectiveness durations.Our methodology is demonstrated by simulations and analysisof a landmark HIV/AIDS clinical trial of short-course nevirapineagainst mother-to-child HIV vertical transmission during labourand delivery.