Random effects modeling approaches for estimating ROC curves from repeated ordinal tests without a gold standard

Estimating diagnostic accuracy without a gold standard is an important problem in medical testing. Although there is a fairly large literature on this problem for the case of repeated binary tests, there is substantially less work for the case of ordinal tests. A noted exception is the work by Zhou, Castelluccio, and Zhou (2005, Biometrics 61, 600-609), which proposed a methodology for estimating receiver operating characteristic (ROC) curves without a gold standard from multiple ordinal tests. A key assumption in their work was that the test results are independent conditional on the true test result. I propose random effects modeling approaches that incorporate dependence between the ordinal tests, and I show through asymptotic results and simulations the importance of correctly accounting for the dependence between tests. These modeling approaches, along with the importance of accounting for the dependence between tests, are illustrated by analyzing the uterine cancer pathology data analyzed by Zhou et al. (2005).     

On the Optimality of F-test on Variance Components in Mixed Linear Models

The analysis-of-variance tests for hypotheses on random effects in regular linear models are considered. Conditions are given for these tests to be uniformly most powerful unbiased or uniformly most powerful invariant unbiased. An example shows that the difference between these conditions can be serious.     

Assessing antiviral potency of anti-HIV therapies in vivo by comparing viral decay rates in viral dynamic models

A virologic marker, the number of HIV RNA copies or viral load, is currently used to evaluate anti-HIV therapies in AIDS clinical trials. This marker can be used to assess the antiviral potency of therapies, but is easily affected by noncompliance, drug resistance, toxicities, and other factors during the long-term treatment evaluation process. Recently it has been suggested to use viral dynamics to assess the potency of antiviral drugs and therapies, since viral decay rates in viral dynamic models have been shown to be related to the antiviral drug potency directly, and they need a shorter evaluation time. In this paper we first review the two statistical approaches for characterizing HIV dynamics and estimating viral decay rates: the individual nonlinear least squares regression (INLS) method and the population nonlinear mixed-effect model (PMEM) approach. To compare the viral decay rates between two treatment arms, parametric and nonparametric tests, based on the estimates of viral decay rates (the derived variables) from both the INLS and PMEM methods, are proposed and studied. We show, using the concept of exchangeability, that the test based on the empirical Bayes' estimates from the PMEM is valid, powerful and robust. This proposed method is very useful in most practical cases where the INLS-based tests and the general likelihood ratio test may not apply. We validate and compare various tests for finite samples using Monte Carlo simulations. Finally, we apply the proposed tests to an AIDS clinical trial to compare the antiviral potency between a 3-drug combination regimen and a 4-drug combination regimen. The proposed tests provide some significant evidence that the 4-drug regimen is more potent than the 3-drug regimen, while the naive methods fail to give a significant result.*To whom correspondence should be addressed.     

An Empirical Power Comparison of Three Tests of Exponentiality Under Mixture- and Outlier-Models

We present the results of an empirical power study of three prominent goodness-of-fit tests for exponentiality due to Shapiro and Wilk (1972), Durbin (1975), and Tiku (1980) by considering the mixture- and the outlier- exponential models as alternatives. This study is on similar lines as those of Dyer and Harbin (1981) and Balakrishnan (1983). We show that Tiku's test is on the whole considerably more powerful than the other two tests.     

Testing models of trophic dynamics: The problem of translating from model to nature

Abstract Mathematical models of interacting populations have a prominent position in population and community ecology, but are often criticized for not being testable. The authors reviewed tests of a particular model, the exploitation ecosystem hypothesis as it was formulated in Oksanen et al. (1981), in order to study problems that may be encountered when testing models. A general problem is how to determine if an experimental system should be regarded as within the model's theoretical domain or not. The theoretical domain defines the type of system the model is meant to apply to. It is noted that both liberal and strict domain definitions can be problematic. Most important is that a too liberal domain definition can result in false understanding (i.e. that it is falsely concluded that the processes included in the model are controlling the study system). Other problems encountered were more system‐specific. Equilibrium predictions were tested in experiments that were too short to reach steady state and in several studies ambiguous definitions and measurements of model variables were found such as productivity, biomass and the number trophic levels. It is concluded that a major obstacle when performing tests is the conceptual and methodological problems encountered when translating model abstractions into an empirical reality.     

Comparing mixed models and Random Forest association tests using naturalgwas and a Striped Bass SNP data set

In this study, we used the phenotype simulation package naturalgwas to test the performance of Zhao's Random Forest method in comparison to an uncorrected Random Forest test, latent factor mixed models (LFMM), genome-wide efficient mixed models (GEMMA), and confounder adjusted linear regression (CATE). We created 400 sets of phenotypes, corresponding to five effect sizes and two, five, 15, or 30 causal loci, simulated from two empirical data sets containing SNPs from Striped Bass representing three and 13 populations. All association methods were evaluated for their ability to detect genotype–phenotype associations based on power, false discovery rates, and number of false positives. Genomic inflation was highest for uncorrected Random Forest and LFMM tests and lowest for Gemma and Zhao's Random Forest. All association tests had similar power to detect causal loci, and Zhao's Random Forest had the lowest false discovery rate in all scenarios. To measure the performance of association tests in small data sets with few loci surrounding a causal gene we also ran analyses again after removing causal loci from each data set. All association tests were only able to find true positives, defined as loci located within 30 kbp of a causal locus, in 3%–18% of simulations. In contrast, at least one false positive was found in 17%–44% of simulations. Zhao's Random Forest again identified the fewest false positives of all association tests studied. The ability to test the power of association tests for individual empirical data sets can be an extremely useful first step when designing a GWAS study.     

The use of generalized linear models and generalized estimating equations in bioarchaeological studies

The current article explores whether the application of generalized linear models (GLM) and generalized estimating equations (GEE) can be used in place of conventional statistical analyses in the study of ordinal data that code an underlying continuous variable, like entheseal changes. The analysis of artificial data and ordinal data expressing entheseal changes in archaeological North African populations gave the following results. Parametric and nonparametric tests give convergent results particularly for P values <0.1, irrespective of whether the underlying variable is normally distributed or not under the condition that the samples involved in the tests exhibit approximately equal sizes. If this prerequisite is valid and provided that the samples are of equal variances, analysis of covariance may be adopted. GLM are not subject to constraints and give results that converge to those obtained from all nonparametric tests. Therefore, they can be used instead of traditional tests as they give the same amount of information as them, but with the advantage of allowing the study of the simultaneous impact of multiple predictors and their interactions and the modeling of the experimental data. However, GLM should be replaced by GEE for the study of bilateral asymmetry and in general when paired samples are tested, because GEE are appropriate for correlated data. Am J Phys Anthropol 153:473–483, 2014. © 2013 Wiley Periodicals, Inc.     

Binary Responses and the Three-period Cross-over

Problems with carry-over effects in the simple two-period cross-over have lead to interest in more complex cross-over designs. A method for analysing the optimum two-treatment three-period design with binary response variables is given by making a simple extension to Gart's logistic model. The method gives independent tests for, and estimates of the difference in treatment and first-order carry-over effects. An example of the analysis is given, using the loglinear models facility in GLIM.     

Improving profile HMM discrimination by adapting transition probabilities

Profile hidden Markov models (HMMs) are used to model protein families and for detecting evolutionary relationships between proteins. Such a profile HMM is typically constructed from a multiple alignment of a set of related sequences. Transition probability parameters in an HMM are used to model insertions and deletions in the alignment. We show here that taking into account unrelated sequences when estimating the transition probability parameters helps to construct more discriminative models for the global/local alignment mode. After normal HMM training, a simple heuristic is employed that adjusts the transition probabilities between match and delete states according to observed transitions in the training set relative to the unrelated (noise) set. The method is called adaptive transition probabilities (ATP) and is based on the HMMER package implementation. It was benchmarked in two remote homology tests based on the Pfam and the SCOP classifications. Compared to the HMMER default procedure, the rate of misclassification was reduced significantly in both tests and across all levels of error rate.     

The effect of time-varying mortality and carbon assimilation on models of carbon allocation in annual plants

Summary Models of optimal carbon allocation schedules have influenced the way plant ecologists think about life history evolution, particularly for annual plants. The present study asks (1) how, within the framework of these models, are their predictions affected by within-season variation in mortality and carbon assimilation rates?; and (2) what are the consequences of these prediction changes for empirical tests of the models? A companion paper examines the basic assumptions of the models themselves. I conducted a series of numerical experiments with a simple carbon allocation model. Results suggest that both qualitative and quantitative predictions can sometimes be sensitive to parameter values for net assimilation rate and mortality: for some parameter values, both the time and size at onset of reproduction, as well as the number of reproductive intervals, vary considerably as a result of small variations in these parameters. For other parameter values, small variations in the parameters result in only small changes in predicted phenotype, but these have very large fitness consequences. Satisfactory empirical tests are thus likely to require much accuracy in parameter estimates. The effort required for parameter estimation imposes a practical constraint on empirical tests, making large multipopulation comparisons impractical. It may be most practical to compare the predicted and observed fitness consequences of variation in the timing of onset of reproduction.     

Robustness of Generalized Estimating Equation (GEE) Tests of Significance against Misspecification of the Error Structure Model

Generalized linear model analyses of repeated measurements typically rely on simplifying mathematical models of the error covariance structure for testing the significance of differences in patterns of change across time. The robustness of the tests of significance depends, not only on the degree of agreement between the specified mathematical model and the actual population data structure, but also on the precision and robustness of the computational criteria for fitting the specified covariance structure to the data. Generalized estimating equation (GEE) solutions utilizing the robust empirical sandwich estimator for modeling of the error structure were compared with general linear mixed model (GLMM) solutions that utilized the commonly employed restricted maximum likelihood (REML) procedure. Under the conditions considered, the GEE and GLMM procedures were identical in assuming that the data are normally distributed and that the variance‐covariance structure of the data is the one specified by the user. The question addressed in this article concerns relative sensitivity of tests of significance for treatment effects to varying degrees of misspecification of the error covariance structure model when fitted by the alternative procedures. Simulated data that were subjected to monte carlo evaluation of actual Type I error and power of tests of the equal slopes hypothesis conformed to assumptions of ordinary linear model ANOVA for repeated measures except for autoregressive covariance structures and missing data due to dropouts. The actual within‐groups correlation structures of the simulated repeated measurements ranged from AR(1) to compound symmetry in graded steps, whereas the GEE and GLMM formulations restricted the respective error structure models to be either AR(1), compound symmetry (CS), or unstructured (UN). The GEE‐based tests utilizing empirical sandwich estimator criteria were documented to be relatively insensitive to misspecification of the covariance structure models, whereas GLMM tests which relied on restricted maximum likelihood (REML) were highly sensitive to relatively modest misspecification of the error correlation structure even though normality, variance homogeneity, and linearity were not an issue in the simulated data.Goodness‐of‐fit statistics were of little utility in identifying cases in which relatively minor misspecification of the GLMM error structure model resulted in inadequate alpha protection for tests of the equal slopes hypothesis. Both GEE and GLMM formulations that relied on unstructured (UN) error model specification produced nonconservative results regardless of the actual correlation structure of the repeated measurements. A random coefficients model produced robust tests with competitive power across all conditions examined. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

Robust Tests for Candidate‐Gene Association Using Case‐Parents Trios with a Multi‐Allelic Marker

In case‐parents trios design, the association between a multi‐allelic candidate‐gene and a disease can be detected by using maximum of score tests (max‐score) when the mode of inheritance is known. We apply the maximum of the max‐score statistics and the maximum of likelihood ratio statistics when the genetic model is unknown and examine their robust properties compared to max‐score statistics. The simulation results demonstrate that the two maximum robust tests are more efficacious and robust across all genetic models compared with the three max‐score tests. Moreover, in most situations, the maximum of the max‐score tests seems to be more powerful than the maximum of the likelihood ratio tests. (© 2004 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)     

A transitional model for longitudinal binary data subject to nonignorable missing data

Binary longitudinal data are often collected in clinical trials when interest is on assessing the effect of a treatment over time. Our application is a recent study of opiate addiction that examined the effect of a new treatment on repeated urine tests to assess opiate use over an extended follow-up. Drug addiction is episodic, and a new treatment may affect various features of the opiate-use process such as the proportion of positive urine tests over follow-up and the time to the first occurrence of a positive test. Complications in this trial were the large amounts of dropout and intermittent missing data and the large number of observations on each subject. We develop a transitional model for longitudinal binary data subject to nonignorable missing data and propose an EM algorithm for parameter estimation. We use the transitional model to derive summary measures of the opiate-use process that can be compared across treatment groups to assess treatment effect. Through analyses and simulations, we show the importance of properly accounting for the missing data mechanism when assessing the treatment effect in our example.     

Null model analyses of presence–absence matrices need a definition of independence

Tests for species interactions that involve the comparison of a statistic calculated from observed matrix of species presences and absences with the distribution of the same statistic generated from a null model have been used by ecologists for about 30 years. We argue that the validity of these tests requires a specific definition of independence. In particular, we note that an assumption that is often made is that all presence–absence matrices with the same row and column totals are equally likely if there is no interaction. However, we show using a simple model for species presences and absences without any species interactions that, in general, this assumption should be made with caution. Our model incorporates a definition of independence, allowing the computation of probabilities of different patterns in the null matrices. Other definitions of independence are possible; one of them is outlined using a new generalized linear model approach for carrying out tests applicable to different null models with or without the assumption of keeping row and column totals fixed. Electronic supplementary material  The online version of this article (doi:) contains supplementary material, which is available to authorized users.     

A mixed two-stage method for detecting interactions in genomewide association studies

Genomewide association studies (GWAS) are being conducted to unravel the genetic etiology of complex diseases, in which complex epistasis may play an important role. One-stage method in which interactions are tested using all samples at one time may be computationally problematic, may have low power as the number of markers tested increases and may not be cost-efficient. A common two-stage method may be a reasonable and powerful approach for detecting interacting genes using all samples in both two stages. In this study, we introduce an alternative two-stage method, in which some promising markers are selected using a proportion of samples in the first stage and interactions are then tested using the remaining samples in the second stage. This two-stage method is called mixed two-stage method. We then investigate the power of both one-stage method and mixed two-stage method to detect interacting disease loci for a range of two-locus epistatic models in a case-control study design. Our results suggest that mixed two-stage method may be more powerful than one-stage method if we choose about 30% of samples for single-locus tests in the first stage, and identify less than and equal to 1% of markers for follow-up interaction tests. In addition, we compare both two-stage methods and find that our two-stage method will lose power because we only use part of samples in both two stages.     

The sensitivity and specificity of markers for event times

The statistical literature on assessing the accuracy of risk factors or disease markers as diagnostic tests deals almost exclusively with settings where the test, Y, is measured concurrently with disease status D. In practice, however, disease status may vary over time and there is often a time lag between when the marker is measured and the occurrence of disease. One example concerns the Framingham risk score (FR-score) as a marker for the future risk of cardiovascular events, events that occur after the score is ascertained. To evaluate such a marker, one needs to take the time lag into account since the predictive accuracy may be higher when the marker is measured closer to the time of disease occurrence. We therefore consider inference for sensitivity and specificity functions that are defined as functions of time. Semiparametric regression models are proposed. Data from a cohort study are used to estimate model parameters. One issue that arises in practice is that event times may be censored. In this research, we extend in several respects the work by Leisenring et al. (1997) that dealt only with parametric models for binary tests and uncensored data. We propose semiparametric models that accommodate continuous tests and censoring. Asymptotic distribution theory for parameter estimates is developed and procedures for making statistical inference are evaluated with simulation studies. We illustrate our methods with data from the Cardiovascular Health Study, relating the FR-score measured at enrollment to subsequent risk of cardiovascular events.     

A Probit Latent Class Model with General Correlation Structures for Evaluating Accuracy of Diagnostic Tests

Summary Traditional latent class modeling has been widely applied to assess the accuracy of dichotomous diagnostic tests. These models, however, assume that the tests are independent conditional on the true disease status, which is rarely valid in practice. Alternative models using probit analysis have been proposed to incorporate dependence among tests, but these models consider restricted correlation structures. In this article, we propose a probit latent class model that allows a general correlation structure. When combined with some helpful diagnostics, this model provides a more flexible framework from which to evaluate the correlation structure and model fit. Our model encompasses several other PLC models but uses a parameter‐expanded Monte Carlo EM algorithm to obtain the maximum‐likelihood estimates. The parameter‐expanded EM algorithm was designed to accelerate the convergence rate of the EM algorithm by expanding the complete‐data model to include a larger set of parameters and it ensures a simple solution in fitting the PLC model. We demonstrate our estimation and model selection methods using a simulation study and two published medical studies.     

A Simulation Study of Permutation,Bootstrap, and Gene Dropping for Assessing Statistical Significance in the Case of Unequal Relatedness

We used simulations to evaluate methods for assessing statistical significance in association studies. When the statistical model appropriately accounted for relatedness among individuals, unrestricted permutation tests and a few other simulation-based methods effectively controlled type I error rates; otherwise, only gene dropping controlled type I error but at the expense of statistical power.     

Tests of proportional hazards and proportional odds models for grouped survival data

In this paper, we derive score test statistics to discriminate between proportional hazards and proportional odds models for grouped survival data. These models are embedded within a power family transformation in order to obtain the score tests. In simple cases, some small-sample results are obtained for the score statistics using Monte Carlo simulations. Score statistics have distributions well approximated by the chi-squared distribution. Real examples illustrate the proposed tests.