Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices

Background

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

Methods and Findings

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

Conclusions

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.     

Biomedical research funding: when the game gets tough, winners start to play

Extramural funding provides major support for biomedical research in academia, and National Institutes of Health (NIH) grants often constitute direct evaluation criteria for promotions and tenure. Therefore, NIH budget trends influence long-term scientific strategies and career decisions, as well as the progress of science itself. Our analysis of the last 37 years of NIH awards, however, reveals that the success rate of grant applications submitted for funding is negatively related to the total yearly amount of (inflation-adjusted) NIH extramural expenditure. Instead, as might be expected, the ratio between available funding and the number of submission directly predicts the probability of winning support in any given year. We purport that the considerable success rate variability can be parsimoniously explained by a proportional but delayed reaction of the number of applications to budget fluctuations. As a counterintuitive consequence, grant proposals conceived during lean periods might stand the best chance of success.     

Use and importance of the NIH noncompeting continuation application

Each year National Institutes of Health (NIH) grant recipients must submit a noncompeting continuation application before receiving continued federal funding. This paper describes the use and value of the application. Investigators benefit by a yearly self-assessment of the research progress and future plans. The noncompeting continuation application is part of the important communication and interaction that should exist between the investigator and NIH staff. NIH staff members use the application to determine important scientific advances that have resulted from supported grants. Many planning activities and required reports are based on information contained in these applications. NIH staff performs scientific and budgetary review to ensure that research progress is satisfactory and that all budgetary and certification issues are in order. Detailed guidance is provided to help the grantee prepare the application. A separate significance section is suggested as a means to document key findings and their importance.     

The effects of adaptation on the perception of similar and dissimilar odors

The Pubishers wish to apologise for the inadvertent mis-spellingof Charles J. Wysocki and other errors in the above article,which should be corrected as follows. On Page 471, Table I, the first Number in the second columnshould read 2:6. On page 477, Table II, the results under d-limonene should readas below. On page 481, the NIH grant number to CJW is DC00298.     

Two strikes: limited NIH R55 and R56 retooling funds and abolishment of the A2 grant mechanism

The U.S. National Institutes of Health (NIH) are facing significant budgetary challenges as a result of the current economic climate. The recent sunset of investigator-initiated R01-type research grants after one revised submission, coupled with the present lack of an NIH retooling funding mechanism for such grant applicants, creates a concerning risk that talented and well-trained investigators may be forced to give up their research careers. Existing NIH retooling mechanisms include the R55 Shannon Award, which was established in 1991 and was essentially replaced in 2005 by the R56 award. There is an urgent need to either significantly expand the R55/R56 mechanisms and definition of NIH grant bridging/retooling support for unfunded meritorious proposals or introduce a new mechanism that provides specific support to investigators with competitive but unfunded R01 revised grants. An expanded retooling funding mechanism deserves implementation during continuing assessment of whether allowance of only one revision of research proposals has achieved its initial intended goals.     

Challenges and opportunities: the expanded government-led HIV/AIDS programs in China

HIV/AIDS has posed an increasingly serious issue in China. In recent years, Chinese government has taken further intensified efforts to combat HIV/AIDS with high-level political commitment, supportive policy development, increased financial allocation, large-scale of government-led initiatives, expanded international cooperation and great involvement of non-governmental organizations. Meanwhile gaps and challenges coexist impacting the implementation and the results of national HIV/AIDS programs. Thus, further government efforts are needed to improve and tailor the actions to meet the requirement of HIV/AIDS control in China. Foundation item: China International Clinical, Operational, and Health Services Research and Training Award with NIH grant number of U2R TW006918.     

The "study" role of past National Institutes of Health study sections

The original National Institutes of Health (NIH) study sections had two missions. The review of grant applications was the enduring one that we all recognize. The second original function, less remembered today, was to stand ready to advise the NIH, and in fact the entire community in a given biomedical field, on the current state of that discipline, as well as to opine on what new vistas were arising and to suggest (or occasionally even launch) appropriate courses of action. The present contribution is intended to remind us of this lesser-known original function of NIH study sections. We might ponder whether today's study sections, although more overworked than Sisyphus, should again take up this second function.     

To Apply or Not to Apply: A Survey Analysis of Grant Writing Costs and Benefits

We surveyed 113 astronomers and 82 psychologists active in applying for federally funded research on their grant-writing history between January, 2009 and November, 2012. We collected demographic data, effort levels, success rates, and perceived non-financial benefits from writing grant proposals. We find that the average proposal takes 116 PI hours and 55 CI hours to write; although time spent writing was not related to whether the grant was funded. Effort did translate into success, however, as academics who wrote more grants received more funding. Participants indicated modest non-monetary benefits from grant writing, with psychologists reporting a somewhat greater benefit overall than astronomers. These perceptions of non-financial benefits were unrelated to how many grants investigators applied for, the number of grants they received, or the amount of time they devoted to writing their proposals. We also explored the number of years an investigator can afford to apply unsuccessfully for research grants and our analyses suggest that funding rates below approximately 20%, commensurate with current NIH and NSF funding, are likely to drive at least half of the active researchers away from federally funded research. We conclude with recommendations and suggestions for individual investigators and for department heads.     

Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France

Background

Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.

Methodology and Principal Findings

We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.

Conclusions

Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.     

Graphic-digitizer analysis of axon spectra in ethmoidal and lingual branches of the trigeminal nerve

Summary Sections were removed from the lingual and ethmoidal nerves of cats and histologically prepared, and the fibers were analyzed under the light microscope. Neural dimensions were measured by a new technique, employing a graphic digitizer and computer. The outline of a neural structure was traced with the digitizer pen, and the total number of axons, their cross-sectional areas, shapes, diameter spectra, and locations within the nerve were calculated.Both nerves had unimodal axon spectra with the peak between 2 and 6 m diameter. Differences in axon composition occurred over the diameter range of 9 to 20 ; the lingual nerve had many axons in this range, the ethmoidal nerve only a few. The total number of myelinated axons was near 4000 in the lingual nerve, near 1400 in the ethmoidal nerve; only the latter had many large-sized Remak bundles (containing C-fibers). Most myelinated axons were not perfectly circular but exhibited various degrees of distortion.This investigation was supported by NIH grants DE 02152 and DE 00248. The use of the graphic digitizer was made possible through NIH grant FR 00374, and the use of the ultramicrotome through NIH grant AM 13182.     

Compartment modeling from tracer experiments

Anelementary system is defined as two compartments, one the immediate and unique precursor of the other. By synthesizing a number of elementary systems, we can obtain a system with any givenweight, transfer time anddispersion. A criterion is presented to choose the best model of a given system, i.e. a system with the same above parameters and composed of a minimum number of elementary systems. Supported in part by NIH grant NO. GM-572. Supported by PHS Training grant No. 2T01 HE05138-17 from the National Heart Institute.     

A method to express complex curves concisely and with high fidelity

Piecewise spline interpolation between judiciously selected points on a curve reproduces it with great precision. A number of points much smaller than in current digitization techniques is needed and their positions fully characterize a curve. We present the rules to select the nodal points and we demonstrate that their positions are very sensitive to changes in wave form. They can therefore be used to identify a curve in parts or in its entirety, to provide its mathematical definition.     

Tutorial on large deviations for the binomial distribution

We present, in an easy to use form, the large deviation theory of the binomial distribution: how to approximate the probability ofk or more successes inn independent trials, each with success probabilityp, when the specified fraction of successes,a≡k/n, satisfies 0<p<a<1. Supported by NIH grant GM 36230 and NSF grant DMS 8601986. Supported by NIH grant GM 36230 and a grant from the System Development Foundation.     

Expression of Nanog gene promotes NIH3T3 cell proliferation

Cells are the functional elements in tissue engineering and regenerative medicine. A large number of cells are usually needed for these purposes. However, there are numbers of limitations for in vitro cell proliferation. Nanog is an important self-renewal determinant in embryonic stem cells. However, it remains unknown whether Nanog will influence the cell cycle and cell proliferation of mature cells. In this study, we expressed Nanog in NIH3T3 cells and showed that expression of Nanog in NIH3T3 promoted cells to enter into S phase and enhanced cell proliferation. This suggests that Nanog gene might function in a similar fashion in mature cells as in ES cells. In addition, it may provide an approach for in vitro cell expansion.     

A study of cultural transmission in Taiwan

Our study of cultural transmission in Taiwan is based on a survey of 1000 students, their families, and friends, for characters ranging from religion to various customs and beliefs, as well as entertainment and hygienic habits. The effects of father, mother, and an older sib on a propositus are tested by an additive model of transmission, using a novel statistical procedure, and compared with correlations with friends. For many traits there exist significant influences; older sibs are almost as important as father and mother, with effects differing somewhat with their sex. Formulas for recurrences and equilibrium frequency of a cultural character for which father, mother, and sib are active in transmission are given in the appendices along with formulas for estimation of their effects from real data.This research was supported in part by grant NIH GM 20467 and NIH GM 20816. In early part of this investigation K.H.C. was supported by a grant to SIMS from the Alfred P. Sloan Foundation.     

Exclusion gene mapping utilizing patients with chromosome imbalance: The HL-A system as a prototype

Summary 17 chromosomally unbalanced patients, their siblings and parents were tested for HL-A types and for up to 25 other polymorphic systems to determine whether there was gain or loss of an allele concurrent with the gain or loss of chromosome material. 5 patients had trisomy of part or all of a chromosome; 2 had trisomy of a segment and also deletion of chromosome material. All 7 were due to a familial translocation. The remaining patients had small deletions; 5 had ring chromosomes, 4 had rod deletions and 1 had missing chromosome material due to a heritable translocation. All cases were informative at the HL-A loci because of the high degree of polymorphism of the system whereas only some of the other systems were informative. None of the 17 patients showed unusual inheritance of HL-A or any other of the polymorphic systems examined. These results provide evidence excluding the HL-A and other loci from a number of possible locations in the human genome.Supported by Maternal and Child Health Services 920 and a grant from the Medical Research Foundation of Oregon.Supported by NIH grants AM 13173 and HD-05082 and a Basil O'Connor Starter Research Grant from the National Foundation March of Dimes.Supported by Crippled Children's Division of Oregon Summer Student Funds.Supported by a grant from the National Foundation March of Dimes.     

Codifying Collegiality: Recent Developments in Data Sharing Policy in the Life Sciences

Over the last decade, there have been significant changes in data sharing policies and in the data sharing environment faced by life science researchers. Using data from a 2013 survey of over 1600 life science researchers, we analyze the effects of sharing policies of funding agencies and journals. We also examine the effects of new sharing infrastructure and tools (i.e., third party repositories and online supplements). We find that recently enacted data sharing policies and new sharing infrastructure and tools have had a sizable effect on encouraging data sharing. In particular, third party repositories and online supplements as well as data sharing requirements of funding agencies, particularly the NIH and the National Human Genome Research Institute, were perceived by scientists to have had a large effect on facilitating data sharing. In addition, we found a high degree of compliance with these new policies, although noncompliance resulted in few formal or informal sanctions. Despite the overall effectiveness of data sharing policies, some significant gaps remain: about one third of grant reviewers placed no weight on data sharing plans in their reviews, and a similar percentage ignored the requirements of material transfer agreements. These patterns suggest that although most of these new policies have been effective, there is still room for policy improvement.     

The future of family practice training in California.

Although the number of physicians in California has doubled since 1963, the number of family and general practice physicians has declined. The ratio of office-based primary care physicians to population has also decreased. Graduate medical education is funded largely from patient care revenues, but the low rate of reimbursement for ambulatory care makes training in primary care specialties especially dependent on public support. Medicare, the Veterans Administration, and the University of California provide more than $325 million a year in support of graduate medical education in California. Federal and state grant programs provide $5 million a year for family physician training in the state, but appropriations to these programs have been reduced in real terms. California family practice residencies are disproportionately located at county hospitals, where funding shortfalls make them especially vulnerable to cuts in grant programs. Additional resources will be needed if more family physicians are to be trained.     

Enhancing NIH grant peer review: a broader perspective

Over the next couple of years, NIH will be revising its process of reviewing grant applications. The planned changes will make the NIH system more similar in some ways to those of European funding agencies, while retaining many unique features.