Examining the Predictive Validity of NIH Peer Review Scores

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.     

Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process.Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.We analyzed reviewer scoring data from a research program that had recently modified the review setting from face-to-face to a teleconference format with minimal changes to the overall review procedures. This analysis included approximately 1600 applications over a 4-year period: two years of face-to-face panel meetings compared to two years of teleconference meetings. The average overall scientific merit scores, score distribution, standard deviations and reviewer inter-rater reliability statistics were measured, as well as reviewer demographics and length of time discussing applications.The data indicate that few differences are evident between face-to-face and teleconference settings with regard to average overall scientific merit score, scoring distribution, standard deviation, reviewer demographics or inter-rater reliability. However, some difference was found in the discussion time.These findings suggest that most review outcome measures are unaffected by review setting, which would support the trend of using teleconference reviews rather than face-to-face meetings. However, further studies are needed to assess any correlations among discussion time, application funding and the productivity of funded research projects.     

The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies

There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period. Peer review scores associated with applications were found to be moderately correlated with the total time-adjusted citation output of funded projects, although a high degree of variability existed in the data. Analysis over time revealed that as average annual scores of all applications (both funded and unfunded) submitted to this program improved with time, the average annual citation output per application increased. Citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget. However, the number of funded applications per year was found to correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be an efficient strategy to optimize the scientific impact of research program portfolios. This strategy must be weighed against the need for a balanced research portfolio and the inherent high costs of some areas of research. The relationship observed between peer review scores and bibliometric output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.     

Biomedical research funding: when the game gets tough, winners start to play

Extramural funding provides major support for biomedical research in academia, and National Institutes of Health (NIH) grants often constitute direct evaluation criteria for promotions and tenure. Therefore, NIH budget trends influence long-term scientific strategies and career decisions, as well as the progress of science itself. Our analysis of the last 37 years of NIH awards, however, reveals that the success rate of grant applications submitted for funding is negatively related to the total yearly amount of (inflation-adjusted) NIH extramural expenditure. Instead, as might be expected, the ratio between available funding and the number of submission directly predicts the probability of winning support in any given year. We purport that the considerable success rate variability can be parsimoniously explained by a proportional but delayed reaction of the number of applications to budget fluctuations. As a counterintuitive consequence, grant proposals conceived during lean periods might stand the best chance of success.     

Use and importance of the NIH noncompeting continuation application

Each year National Institutes of Health (NIH) grant recipients must submit a noncompeting continuation application before receiving continued federal funding. This paper describes the use and value of the application. Investigators benefit by a yearly self-assessment of the research progress and future plans. The noncompeting continuation application is part of the important communication and interaction that should exist between the investigator and NIH staff. NIH staff members use the application to determine important scientific advances that have resulted from supported grants. Many planning activities and required reports are based on information contained in these applications. NIH staff performs scientific and budgetary review to ensure that research progress is satisfactory and that all budgetary and certification issues are in order. Detailed guidance is provided to help the grantee prepare the application. A separate significance section is suggested as a means to document key findings and their importance.     

It Is Time to Re‐Evaluate the Peer Review Process for Preclinical Research

Problems in peer review, the backbone of maintaining high standards in scientific publishing, have led to wide spread discontent within the scientific community. Training in the peer review process and a simpler format to assist in decision making are possible courses to improve and expedite the process of peer review and scientific publishing.     

An analysis of preliminary and post-discussion priority scores for grant applications peer reviewed by the Center for Scientific Review at the NIH

There has been the impression amongst many observers that discussion of a grant application has little practical impact on the final priority scores. Rather the final score is largely dictated by the range of preliminary scores given by the assigned reviewers. The implication is that the preliminary and final scores are the same and the discussion has little impact. The purpose of this examination of the peer review process at the National Institutes of Health is to describe the relationship between preliminary priority scores of the assigned reviewers and the final priority score given by the scientific review group. This study also describes the practical importance of any differences in priority scores. Priority scores for a sample of standard (R01) research grant applications were used in this assessment. The results indicate that the preliminary meeting evaluation is positively correlated with the final meeting outcome but that they are on average significantly different. The results demonstrate that discussion at the meeting has an important practical impact on over 13% of the applications.     

A view from the NIH bridge: perspectives of a program officer

This essay is written from my perspective as a program officer for research and training activities at the National Institute of General Medical Sciences (NIGMS) for almost 27 yr. It gives a bird's-eye view of the job of a program officer, which includes providing advice to applicants and grantees, making funding recommendations, overseeing grantees' progress, facilitating scientific opportunities in specific areas of program responsibility, and shaping NIGMS and National Institutes of Health (NIH) policy. I have highlighted the numerous rewards of serving as a program officer, as well as some of the difficulties. For those who may be considering a position as an NIH program officer now or in the future, I've also described the qualities and qualifications that are important for such a career choice. Finally, this essay addresses some of the challenges for the NIH and the research community in the years ahead as we simultaneously face exciting scientific opportunities and tighter budgets.     

Will technology trample peer review in ecology? Ongoing issues and potential solutions

The classical view of peer review is that it is our primary process for assessing and judging whether research results should be published in a scholarly journal. However, the increased pressure to publish and technological developments are transforming peer review such that it is becoming a system that judges where work is published rather than whether the research is publishable (a ‘where rather than if’ process). Ecology is a field in which publication numbers puts a particular pressure on the review system. In this forum piece, I summarize the issues with the current publication system and discuss how technology is changing it, while suggesting solutions for important prior and ongoing issues with the peer review system. The view explored here is that technological developments (e.g. ease of creating journals, internet sites, storage, data generation, sharing of data and analytical code) will not eliminate peer review per se but will allow for a new set of parameters in which ethics and the optimal use of public funding will play a vital role in the evolution of the review process. Synthesis The number of papers and journals in Ecology has increased dramatically in the past decade. I present a critical overview of our review system and proposes that pressure to publish and technological developments have transformed peer review into a system that decides “where rather than if” papers are publishable. While reviewing the current pressures and factors playing a vital role in the evolution of the review and publication systems, I propose potential solutions to deal with current and future challenges to the peer review and publication systems.     

Prescriptions for medical research. I--Management within the Medical Research Council.

In their submission to the government in advance of the white paper on science policy in the United Kingdom the Medical Research Council commends the MRC''s own approach to managing directly funded research. But a series of semi-structured interviews with the directors of some of the MRC''s units suggests a gap between the MRC''s model of managed research and the reality. Although such units are theoretically managed from MRC head office (and units are charged an overhead for this), in practice each unit runs its own affairs. Between major reviews average contact time with the head office contact person is seven hours a year. The first paper argues that a purchaser-provider split would recognise the benefits of decentralisation and allow units to bid for research funds from several sources, the successful ones guaranteeing their survival through a rolling series of research programmes. The second paper criticises the MRC''s cumbersome peer review system. Reliance on outside experts atrophies the scientific skills of head office staff and builds delays into decision making. A purchaser-provider model would allow the head office scientific staff to act like commercial research and development managers, commissioning research, and using the outcome, rather than peer review, as a criterion for continued funding.     

The National Institutes of Health system for enhancing the science,safety, and ethics of recombinant DNA research

Oversight of recombinant DNA research by the National Institutes of Health (NIH) is predicated on ethical and scientific responsibilities that are akin, in many ways, to those that pertain to the oversight of animal research. The NIH system of oversight, which originated more than 25 years ago, is managed by the NIH Office of Biotechnology Activities (OBA), which uses various tools to fulfill its oversight responsibilities. These tools include the NIH Guidelines for Research Involving Recombinant DNA Molecules (NIH Guidelines) and the Recombinant DNA Advisory Committee. The OBA also undertakes special initiatives to promote the analysis and dissemination of information key to our understanding of recombinant DNA, and in particular, human gene transfer research. These initiatives include a new query-capable database, an analytical board of scientific and medical experts, and conferences and symposia on timely scientific, safety, and policy issues. Veterinary scientists can play an important role in the oversight of recombinant DNA research and in enhancing our understanding of the many safety and scientific dimensions of the field. These roles include developing appropriate animal models, reporting key safety data, enhancing institutional biosafety review, and promoting compliance with the NIH Guidelines.     

The evolution of peer review as a basis for scientific publication: directional selection towards a robust discipline?

Peer review is pivotal to science and academia, as it represents a widely accepted strategy for ensuring quality control in scientific research. Yet, the peer‐review system is poorly adapted to recent changes in the discipline and current societal needs. We provide historical context for the cultural lag that governs peer review that has eventually led to the system's current structural weaknesses (voluntary review, unstandardized review criteria, decentralized process). We argue that some current attempts to upgrade or otherwise modify the peer‐review system are merely sticking‐plaster solutions to these fundamental flaws, and therefore are unlikely to resolve them in the long term. We claim that for peer review to be relevant, effective, and contemporary with today's publishing demands across scientific disciplines, its main components need to be redesigned. We propose directional changes that are likely to improve the quality, rigour, and timeliness of peer review, and thereby ensure that this critical process serves the community it was created for.     

Leadership and innovation. How consensus management blocks genuine innovation

The consensual basis for selecting research topics works for incremental innovation, but is the enemy of radical breakthroughs. Consensual processes, from the ‘European Year of Creativity and Innovation’ to peer review, should be supplemented by the leadership of those with the courage to back the extremes. Such leadership requires support from the very top of scientific management and funding, in academia and industry.     

Sample size and precision in NIH peer review

The Working Group on Peer Review of the Advisory Committee to the Director of NIH has recommended that at least 4 reviewers should be used to assess each grant application. A sample size analysis of the number of reviewers needed to evaluate grant applications reveals that a substantially larger number of evaluators are required to provide the level of precision that is currently mandated. NIH should adjust their peer review system to account for the number of reviewers needed to provide adequate precision in their evaluations.     

Reconceptualising the meaning of scientific communication: open access

The recent US law (H.R.2764) affecting NIH policy and the recent unanimous vote by the Arts and Science faculty of Harvard University in favour of a mandatory deposit of researchers' publications in a suitable repository have brought the Open Access movement into public light. After reviewing the historical background of Open Access, its evolution and extension in the United States, Great Britain, France and Canada are examined. Policies aiming at strengthening Open Access to scientific research are viewed as the direct consequence of treating scientific publishing as an integral part of the research cycle. It should, therefore, be wrapped into the financing of research. As the greater part of research is funded by public money, it appears legitimate to make its results as widely available as is possible. Open Access journals and repositories with strong deposit mandates form the backbone of the strategies to achieve the objective of Open Access. Despite the claims of some publishers, Open Access does not weaken or threaten the peer review process, and it does not conflict with copyright laws.     

Context-Aware Reviewer Assignment for Trust Enhanced Peer Review

Reviewer assignment is critical to peer review systems, such as peer-reviewed research conferences or peer-reviewed funding applications, and its effectiveness is a deep concern of all academics. However, there are some problems in existing peer review systems during reviewer assignment. For example, some of the reviewers are much more stringent than others, leading to an unfair final decision, i.e., some submissions (i.e., papers or applications) with better quality are rejected. In this paper, we propose a context-aware reviewer assignment for trust enhanced peer review. More specifically, in our approach, we first consider the research area specific expertise of reviewers, and the institution relevance and co-authorship between reviewers and authors, so that reviewers with the right expertise are assigned to the corresponding submissions without potential conflict of interest. In addition, we propose a novel cross-assignment paradigm, and reviewers are cross-assigned in order to avoid assigning a group of stringent reviewers or a group of lenient reviewers to the same submission. More importantly, on top of them, we propose an academic CONtext-aware expertise relevanCe oriEnted Reviewer cross-assignmenT approach (CONCERT), which aims to effectively estimate the “true” ratings of submissions based on the ratings from all reviewers, even though no prior knowledge exists about the distribution of stringent reviewers and lenient reviewers. The experiments illustrate that compared with existing approaches, our proposed CONCERT approach can less likely assign more than one stringent reviewers or lenient reviewers to a submission simultaneously and significantly reduce the influence of ratings from stringent reviewers and lenient reviewers, leading to trust enhanced peer review and selection, no matter what kind of distributions of stringent reviewers and lenient reviewers are.     

The Problem of Biased Data and Potential Solutions for Health and Environmental Assessments

The utility and credibility of environmental assessments depend on the use of unbiased data. However, it is increasingly clear that, despite peer review, much of the scientific literature is biased. Sources of bias include publication practices, research design and implementation, funding influences, investigator expectations, statistical methods, confounding, suppression, and fraud. Assessors can take precautions against biased data by performing their own reviews of the sources of data, checking for retractions and corrections, requiring full disclosure of methods, acquiring original data and reanalyzing it, avoiding secondary sources, avoiding unreplicated studies or studies that are not concordant with related studies, and checking for funding or investigator biases. Journals, government agencies, and other institutions can take many more types of actions to reduce bias in scientific data. Some of these are already being implemented but others will require a greater willingness to enforce scientific ethical standards.