Power and Predictive Accuracy of Polygenic Risk Scores

Polygenic scores have recently been used to summarise genetic effects among an ensemble of markers that do not individually achieve significance in a large-scale association study. Markers are selected using an initial training sample and used to construct a score in an independent replication sample by forming the weighted sum of associated alleles within each subject. Association between a trait and this composite score implies that a genetic signal is present among the selected markers, and the score can then be used for prediction of individual trait values. This approach has been used to obtain evidence of a genetic effect when no single markers are significant, to establish a common genetic basis for related disorders, and to construct risk prediction models. In some cases, however, the desired association or prediction has not been achieved. Here, the power and predictive accuracy of a polygenic score are derived from a quantitative genetics model as a function of the sizes of the two samples, explained genetic variance, selection thresholds for including a marker in the score, and methods for weighting effect sizes in the score. Expressions are derived for quantitative and discrete traits, the latter allowing for case/control sampling. A novel approach to estimating the variance explained by a marker panel is also proposed. It is shown that published studies with significant association of polygenic scores have been well powered, whereas those with negative results can be explained by low sample size. It is also shown that useful levels of prediction may only be approached when predictors are estimated from very large samples, up to an order of magnitude greater than currently available. Therefore, polygenic scores currently have more utility for association testing than predicting complex traits, but prediction will become more feasible as sample sizes continue to grow.     

P-Values as Measures of Predictive Validity

A predictive P-value is proposed to measure the difference between an actual and predicted outcome in assessing the validity of an hypothesized prediction model. The concept is illustrated by applications to multiple regression prediction and to the validation of forecast models.     

Predictive Validity of Discriminant Analysis for Genetic Data

We examined the predictive validity of the results using discriminant analysis to distinguish statistically among two or more populations with a large sample of random amplified polymorphic DNA (RAPD) loci, but a small sample of genotypes from each population. We compared and contrasted results from randomized data with results from real data of three studies by 100 randomized shuffling of genotypes into various populations. We generally obtained substantial differences between results from randomized data compared to those from the real data in several characteristics of discriminant analysis. We showed that a high level of correctly classified percentage is also obtainable in the analysis of randomized data, mainly with a low number of populations. However, the correctly classified percentage obtained from the real data was generally significantly higher than the percentage obtained from the randomized data. We suggested that the high level of real differences in allele frequencies of the RAPD polymorphic loci clearly distinguished the various populations and that the populations differ significantly in their RAPD contents in accordance with ecological heterogeneity. We obtained either no or a low level of difference between the correct classification rate obtained by the leaving-one-out procedure and that obtained from the original data, attributed to a low number of loci selected by the stepwise method. The results strengthen and support our conclusion and lead us to focus on the discriminant analysis by selecting only low numbers of discriminating variables.     

Age-Correction of Test Scores Reduces the Validity of Mild Cognitive Impairment in Predicting Progression to Dementia

Objectives

A phase of mild cognitive impairment (MCI) precedes most forms of neurodegenerative dementia. Many definitions of MCI recommend the use of test norms to diagnose cognitive impairment. It is, however, unclear whether the use of norms actually improves the detection of individuals at risk of dementia. Therefore, the effects of age- and education-norms on the validity of test scores in predicting progression to dementia were investigated.

Methods

Baseline cognitive test scores (Syndrome Short Test) of dementia-free participants aged ≥65 were used to predict progression to dementia within three years. Participants were comprehensively examined one, two, and three years after baseline. Test scores were calculated with correction for (1) age and education, (2) education only, (3) age only and (4) without correction. Predictive validity was estimated with Cox proportional hazard regressions. Areas under the curve (AUCs) were calculated for the one-, two-, and three-year intervals.

Results

82 (15.3%) of initially 537 participants, developed dementia. Model coefficients, hazard ratios, and AUCs of all scores were significant (p<0.001). Predictive validity was the lowest with age-corrected scores (−2 log likelihood  = 840.90, model fit χ² (1)  = 144.27, HR  = 1.33, AUCs between 0.73 and 0.87) and the highest with education-corrected scores (−2 log likelihood  = 815.80, model fit χ² (1)  = 171.16, HR  = 1.34, AUCs between 0.85 and 0.88).

Conclusion

The predictive validity of test scores is markedly reduced by age-correction. Therefore, definitions of MCI should not recommend the use of age-norms in order to improve the detection of individuals at risk of dementia.     

Context-Aware Reviewer Assignment for Trust Enhanced Peer Review

Reviewer assignment is critical to peer review systems, such as peer-reviewed research conferences or peer-reviewed funding applications, and its effectiveness is a deep concern of all academics. However, there are some problems in existing peer review systems during reviewer assignment. For example, some of the reviewers are much more stringent than others, leading to an unfair final decision, i.e., some submissions (i.e., papers or applications) with better quality are rejected. In this paper, we propose a context-aware reviewer assignment for trust enhanced peer review. More specifically, in our approach, we first consider the research area specific expertise of reviewers, and the institution relevance and co-authorship between reviewers and authors, so that reviewers with the right expertise are assigned to the corresponding submissions without potential conflict of interest. In addition, we propose a novel cross-assignment paradigm, and reviewers are cross-assigned in order to avoid assigning a group of stringent reviewers or a group of lenient reviewers to the same submission. More importantly, on top of them, we propose an academic CONtext-aware expertise relevanCe oriEnted Reviewer cross-assignmenT approach (CONCERT), which aims to effectively estimate the “true” ratings of submissions based on the ratings from all reviewers, even though no prior knowledge exists about the distribution of stringent reviewers and lenient reviewers. The experiments illustrate that compared with existing approaches, our proposed CONCERT approach can less likely assign more than one stringent reviewers or lenient reviewers to a submission simultaneously and significantly reduce the influence of ratings from stringent reviewers and lenient reviewers, leading to trust enhanced peer review and selection, no matter what kind of distributions of stringent reviewers and lenient reviewers are.     

Researcher Perspectives on Publication and Peer Review of Data

Data “publication” seeks to appropriate the prestige of authorship in the peer-reviewed literature to reward researchers who create useful and well-documented datasets. The scholarly communication community has embraced data publication as an incentive to document and share data. But, numerous new and ongoing experiments in implementation have not yet resolved what a data publication should be, when data should be peer-reviewed, or how data peer review should work. While researchers have been surveyed extensively regarding data management and sharing, their perceptions and expectations of data publication are largely unknown. To bring this important yet neglected perspective into the conversation, we surveyed ∼ 250 researchers across the sciences and social sciences– asking what expectations“data publication” raises and what features would be useful to evaluate the trustworthiness, evaluate the impact, and enhance the prestige of a data publication. We found that researcher expectations of data publication center on availability, generally through an open database or repository. Few respondents expected published data to be peer-reviewed, but peer-reviewed data enjoyed much greater trust and prestige. The importance of adequate metadata was acknowledged, in that almost all respondents expected data peer review to include evaluation of the data’s documentation. Formal citation in the reference list was affirmed by most respondents as the proper way to credit dataset creators. Citation count was viewed as the most useful measure of impact, but download count was seen as nearly as valuable. These results offer practical guidance for data publishers seeking to meet researcher expectations and enhance the value of published data.     

Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process.Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.We analyzed reviewer scoring data from a research program that had recently modified the review setting from face-to-face to a teleconference format with minimal changes to the overall review procedures. This analysis included approximately 1600 applications over a 4-year period: two years of face-to-face panel meetings compared to two years of teleconference meetings. The average overall scientific merit scores, score distribution, standard deviations and reviewer inter-rater reliability statistics were measured, as well as reviewer demographics and length of time discussing applications.The data indicate that few differences are evident between face-to-face and teleconference settings with regard to average overall scientific merit score, scoring distribution, standard deviation, reviewer demographics or inter-rater reliability. However, some difference was found in the discussion time.These findings suggest that most review outcome measures are unaffected by review setting, which would support the trend of using teleconference reviews rather than face-to-face meetings. However, further studies are needed to assess any correlations among discussion time, application funding and the productivity of funded research projects.