Examining the Predictive Validity of NIH Peer Review Scores

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.     

Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices

Background

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

Methods and Findings

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

Conclusions

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.     

Sample size and precision in NIH peer review

The Working Group on Peer Review of the Advisory Committee to the Director of NIH has recommended that at least 4 reviewers should be used to assess each grant application. A sample size analysis of the number of reviewers needed to evaluate grant applications reveals that a substantially larger number of evaluators are required to provide the level of precision that is currently mandated. NIH should adjust their peer review system to account for the number of reviewers needed to provide adequate precision in their evaluations.     

Reliability of a structured method of selecting abstracts for a plastic surgical scientific meeting

There is no generally accepted method for assessing abstracts that are submitted for a medical scientific meeting. This article describes the development and prospective evaluation of such a method applied to the 220 abstracts submitted for the 2000 Annual Meeting of the European Association of Plastic Surgeons. Structured abstracts were evaluated in three categories: aesthetic surgery, basic research, and clinical study. Each anonymous abstract was assessed separately by 10 reputable European plastic surgeons. These reviewers used a structured rating questionnaire which resulted in a score given by each reviewer to each abstract between -6 and +6. The scores of all 10 reviewers were added for each abstract, and the papers were accepted in each of the three categories on the basis of this abridged score. To evaluate the reliability of this structured method of selection, the interrater agreement among the reviewers was tested by means of kappa analysis and the Cronbach alpha coefficient. The kappa values for agreement among reviewers regarding acceptability of abstracts were low, but the alpha coefficient indicated an acceptable degree of reliability of the average reviewers' ratings for all categories. Using a structured questionnaire can be helpful in the objective assessment of abstracts for a scientific meeting and may facilitate comparison of abstracts. Meritocratic dichotomy of abstracts by the reviewers is advocated to further improve reliability of the rating. Even though reliability generally increases with the number of reviewers, the annual increase of submitted abstracts may necessitate a decrease in the number of reviewers for each abstract.     

Context-Aware Reviewer Assignment for Trust Enhanced Peer Review

Reviewer assignment is critical to peer review systems, such as peer-reviewed research conferences or peer-reviewed funding applications, and its effectiveness is a deep concern of all academics. However, there are some problems in existing peer review systems during reviewer assignment. For example, some of the reviewers are much more stringent than others, leading to an unfair final decision, i.e., some submissions (i.e., papers or applications) with better quality are rejected. In this paper, we propose a context-aware reviewer assignment for trust enhanced peer review. More specifically, in our approach, we first consider the research area specific expertise of reviewers, and the institution relevance and co-authorship between reviewers and authors, so that reviewers with the right expertise are assigned to the corresponding submissions without potential conflict of interest. In addition, we propose a novel cross-assignment paradigm, and reviewers are cross-assigned in order to avoid assigning a group of stringent reviewers or a group of lenient reviewers to the same submission. More importantly, on top of them, we propose an academic CONtext-aware expertise relevanCe oriEnted Reviewer cross-assignmenT approach (CONCERT), which aims to effectively estimate the “true” ratings of submissions based on the ratings from all reviewers, even though no prior knowledge exists about the distribution of stringent reviewers and lenient reviewers. The experiments illustrate that compared with existing approaches, our proposed CONCERT approach can less likely assign more than one stringent reviewers or lenient reviewers to a submission simultaneously and significantly reduce the influence of ratings from stringent reviewers and lenient reviewers, leading to trust enhanced peer review and selection, no matter what kind of distributions of stringent reviewers and lenient reviewers are.     

The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period. Peer review scores associated with applications were found to be moderately correlated with the total time-adjusted citation output of funded projects, although a high degree of variability existed in the data. Analysis over time revealed that as average annual scores of all applications (both funded and unfunded) submitted to this program improved with time, the average annual citation output per application increased. Citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget. However, the number of funded applications per year was found to correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be an efficient strategy to optimize the scientific impact of research program portfolios. This strategy must be weighed against the need for a balanced research portfolio and the inherent high costs of some areas of research. The relationship observed between peer review scores and bibliometric output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.     

Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process.Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.We analyzed reviewer scoring data from a research program that had recently modified the review setting from face-to-face to a teleconference format with minimal changes to the overall review procedures. This analysis included approximately 1600 applications over a 4-year period: two years of face-to-face panel meetings compared to two years of teleconference meetings. The average overall scientific merit scores, score distribution, standard deviations and reviewer inter-rater reliability statistics were measured, as well as reviewer demographics and length of time discussing applications.The data indicate that few differences are evident between face-to-face and teleconference settings with regard to average overall scientific merit score, scoring distribution, standard deviation, reviewer demographics or inter-rater reliability. However, some difference was found in the discussion time.These findings suggest that most review outcome measures are unaffected by review setting, which would support the trend of using teleconference reviews rather than face-to-face meetings. However, further studies are needed to assess any correlations among discussion time, application funding and the productivity of funded research projects.     

Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France

Background

Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.

Methodology and Principal Findings

We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.

Conclusions

Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.     

Commissioning a national programme of research and development on the interface between primary and secondary care.

The first call for applications to the NHS research and development programme on the interface between primary and secondary care was advertised in February 1994. A total of 674 outline proposals were submitted and 54 (8%) secured funding. Projects have been commissioned in 16 of the 21 priority areas and around 6m pounds has been committed. Analysis shows that multidisciplinary applications are more likely to be funded and that the odds for a successful application are on average nearly doubled for each discipline represented up to five. A survey of applicants and peer reviewers found satisfaction with much of the commissioning process, but peer review and feedback were subject to criticism, particularly by unsuccessful applicants. The programme shows that it is possible to commission a large number of projects in an innovative area of research and development and has identified refinements that will further increase the efficiency and acceptability of the process.     

Legislation for trial registration and data transparency

Sharing of final research data from clinical research is an essential part of the scientific method. The U.S. National Institutes of Health require some grant applications to include plans for sharing final research data, which it defines as the factual materials necessary to document, support, and validate research findings. In the U.S., however, the Privacy Rule adopted under the Health Insurance Portability and Accountability Act impedes the sharing of final research data. In most situations, final research data may be shared only where all information that could possibly be used to identify the subject has been deleted, or where the subject has given authorization for specific research, or an Institutional Review Board has granted a waiver.     

Verification of a REACH Environmental Prioritization System Against Regulatory Risk Indices

A prioritization method that was developed to rank chemical substances on the basis of their environmental impact was applied to 230 new substance notifications from earlier European chemical legislation (67/548/EEC). The method encompasses three steps: (1) assigning an environmental hazard score, (2) assigning an environmental exposure score, and (3) combining these two scores into one priority score. In this study, the resulting scores ranged between 4 (highest priority) and 15 (lowest priority). The scores were compared with results from risk assessments available for 138 of the 230 substances. For most substances in these assessments, a priority score of 12 or higher was associated with no or limited risk, while a score of 11 or lower represented high risk or led to the conclusion that risk reduction measures were required. This categorization applied to all but 15 of the 138 substances. The method was also used for ranking the first 15 Substances of Very High Concern (SVHC) under REACH regulation, to compare the priority scores of new substances to those of the SVHC. In sum, the prioritization method seems to be valuable to identify substances of concern with respect to the environment.     

A note on the diffusion approximation for single neuron firing problems

Summary This paper presents a well-known stochastic model used to describe the firing or discharge pattern of a single neuron in terms of various input processes, and shows how the potential level of the neuron can be given by means of a diffusion equation approximation. There is a discussion of the adequacy of this approximation, and the paper concludes with a brief discussion of first passage time problems.Supported in part by a grant from the Alfred P. Sloan Foundation to the Committee on Mathematical Biology and by a grant of the Statistical Branch, Office of Naval Research to the Department of Statistics, University of Chicago.     

The flexizyme system: a highly flexible tRNA aminoacylation tool for the translation apparatus

Flexizymes are de novo ribozymes capable of charging a wide variety of non-natural amino acids on tRNAs. The flexizyme system enables reprogramming of the genetic code by reassigning the codons that are generally assigned to natural amino acids to non-natural residues, and thus mRNA-directed synthesis of non-natural polypeptides can be achieved. In this review, we comprehensively summarize the history of the flexizyme system and its subsequent development into a practical tool. Furthermore, applications to the synthesis of novel biopolymers via genetic code reprogramming and perspectives for future applications are described.     

Identifying priority areas for conservation: a global assessment for forest-dependent birds

Limited resources are available to address the world's growing environmental problems, requiring conservationists to identify priority sites for action. Using new distribution maps for all of the world's forest-dependent birds (60.6% of all bird species), we quantify the contribution of remaining forest to conserving global avian biodiversity. For each of the world's partly or wholly forested 5-km cells, we estimated an impact score of its contribution to the distribution of all the forest bird species estimated to occur within it, and so is proportional to the impact on the conservation status of the world's forest-dependent birds were the forest it contains lost. The distribution of scores was highly skewed, a very small proportion of cells having scores several orders of magnitude above the global mean. Ecoregions containing the highest values of this score included relatively species-poor islands such as Hawaii and Palau, the relatively species-rich islands of Indonesia and the Philippines, and the megadiverse Atlantic Forests and northern Andes of South America. Ecoregions with high impact scores and high deforestation rates (2000-2005) included montane forests in Cameroon and the Eastern Arc of Tanzania, although deforestation data were not available for all ecoregions. Ecoregions with high impact scores, high rates of recent deforestation and low coverage by the protected area network included Indonesia's Seram rain forests and the moist forests of Trinidad and Tobago. Key sites in these ecoregions represent some of the most urgent priorities for expansion of the global protected areas network to meet Convention on Biological Diversity targets to increase the proportion of land formally protected to 17% by 2020. Areas with high impact scores, rapid deforestation, low protection and high carbon storage values may represent significant opportunities for both biodiversity conservation and climate change mitigation, for example through Reducing Emissions from Deforestation and Forest Degradation (REDD+) initiatives.     

对基金项目开展网络同行评议的几点思考

国家自然科学基金委生命科学部六处(畜牧兽医与水产和动物学科)通过对2003年基金项目开展网上评议,使我们真正体会到网上评议的优越性,不仅大大提高工作效率,还使同行专家评审意见全文反馈终于成为可能。但是,若要使网上评议更加科学、公正,除了进一步完善专家库外,国家自然科学基金委还需建设一个“基金项目评议的信息资源链接平台”和“专家评议的反评估网络数据库”,从而完善同行评议网络体系。     

Nuffield A-level Biological Science Project

School teachers need help in choosing microscopes for their pupils' use. Suggested specifications are given for various applications. A discussion of the ways of meeting these specifications in practice, is illustrated by current models from British and foreign manufacturers.