Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France

Background

Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers’ and applicants’ perceptions and experiences of CoIs in reviews of French academic grant applications.

Methodology and Principal Findings

We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.

Conclusions

Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.     

Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices

Background

Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.

Methods and Findings

We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.

Conclusions

Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.     

Review of quantitative and qualitative studies on U.S. public perceptions of synthetic biology

How are public perceptions towards synthetic biology likely to evolve? Which factors will impact the framing of this emerging technology, its benefits and risks? The objective of this article is not to draw exhaustive conclusions about public perceptions of synthetic biology, but rather to provide readers with a review of integrated findings from the first quantitative and qualitative research ever conducted on this subject in the United States. Synthetic biology survey research shows two clear findings. The first is that most people know little or nothing about synthetic biology. Second, notwithstanding this lack of knowledge, respondents are likely to venture some remark about what they think synthetic biology is and the tradeoff between potential benefits and potential risks. Finding only some support for the “familiarity argument”—according to which support for emerging technologies will likely increase as awareness of them develops—this article suggests that analogs to cloning, genetic engineering and stem cell research appear to be recurrent in the framing process of synthetic biology. The domain of application seems to be another decisive factor in the framing of synthetic biology. Finally, acceptance of the risk-benefit tradeoff of synthetic biology seems to depend on having an oversight structure that would prove able to manage unknowns, human and environmental concerns, and long-term effects. The most important conclusion of this study is the need for additional investigation of factors that will shape public perceptions about synthetic biology, its potential benefits, and its potential risks.     

Designing and implementing a research integrity promotion plan: Recommendations for research funders

Various stakeholders in science have put research integrity high on their agenda. Among them, research funders are prominently placed to foster research integrity by requiring that the organizations and individual researchers they support make an explicit commitment to research integrity. Moreover, funders need to adopt appropriate research integrity practices themselves. To facilitate this, we recommend that funders develop and implement a Research Integrity Promotion Plan (RIPP). This Consensus View offers a range of examples of how funders are already promoting research integrity, distills 6 core topics that funders should cover in a RIPP, and provides guidelines on how to develop and implement a RIPP. We believe that the 6 core topics we put forward will guide funders towards strengthening research integrity policy in their organization and guide the researchers and research organizations they fund.

Research funders are prominently placed to foster research integrity by requiring that researchers make an explicit commitment to research integrity. This Consensus View suggests 6 core topics that funders should cover in a research integrity promotion plan and provides practical recommendations for how to implement one.

To improve research quality and validity, foster responsible research cultures, and maintain public trust in science, various stakeholders have put research integrity high on their agenda. Among them, research funders are increasingly acknowledging their pivotal role in contributing to a culture of research integrity. For example, the European Commission (EC) is mandating research organizations receiving funding from the €95 billion Horizon Europe program to have, at the institutional level, policies and processes in place for research integrity covering the promotion of good practice, prevention of misconduct and questionable practices, and procedures to deal with breaches of research integrity [1]. To meet these obligations, the EC requires beneficiaries to respect the principles of research integrity as set out in the European Code of Conduct for Research Integrity (ECoC) and suggests that research organizations develop and implement a Research Integrity Promotion Plan (RIPP) [2]. In this Consensus View, we have adopted the World Conference on Research Integrity’s approach to research integrity, by having “research integrity” refer to “the principles and standards that have the purpose to ensure validity and trustworthiness of research” [3]. More specifically, we mostly adhere to the principles outlined in the ECoC: reliability, honesty, respect, and accountability. While many definitions of research integrity exist [4,5], for example, those that distinguish between the integrity of a researcher, integrity of research, and integrity of the research record, the ECoC combines these approaches in a balanced way [1].We believe that funders are prominently placed to foster a culture of research integrity by requiring that the organizations and individual researchers they support make an explicit commitment to research integrity. At the same time, funders need to adopt appropriate research integrity practices themselves. Of late, attention to research integrity among funders has gathered pace, as reflected in several initiatives around the globe that demonstrate how funders can support a culture of research integrity. For example, the US National Science Foundation (NSF) [6] requires applicants’ research organizations to provide training and oversight in the responsible conduct of research, designate individuals responsible for research integrity, and have an institutional certification to testify of its commitment. Also, in 2016, 3 Canadian federal funders joined forces to support research integrity in the Canadian Tri-Agency Framework: Responsible Conduct of Research–Harmony and Responsibility [7]. The framework was subsequently updated in 2021. This framework sets out responsible practices that research organizations and researchers should follow, including rigor, record keeping, accurate referencing, responsible authorship, and the management of conflicts of interest. It also acknowledges the responsibilities of the funders, including “helping to promote the responsible conduct of research and to assist individuals and institutions with the interpretation or implementation of this Responsible Conduct of Research (RCR) Framework”.It is not only major funding organizations in highly developed research environments that are taking steps. Smaller funders are also acting to mandate compliance with research integrity standards. The constantly growing literature on the topic is another sign of development within this area [2,3]. In the USA, research integrity recently reached the political arena, when, following a call from researchers [8], President Biden’s administration published a memorandum on restoring trust [9] that highlights the importance of integrity in research. The memorandum will be supported by the reintroduction of the Scientific Integrity Act. This act will prohibit research misconduct and the manipulation of research findings. It talks of a “culture of research integrity” and demands that funding agencies adopt and enforce research integrity policies, appoint a research integrity officer, and provide regular research integrity and ethics training. The US are not alone in their endeavors. Governments in other countries are equally gearing up to support the integrity and reproducibility of research [10]. However, so far, there is only limited evidence about the effectiveness of such initiatives, although it is generally accepted that they raise awareness among various stakeholders concerning research integrity challenges, strengthen the sense of responsibility of those stakeholders to address those challenges, and thereby ultimately contribute to fostering a culture of research integrity.In a collective effort to foster research integrity, research organizations and funders have their own, complementary roles. The Standard Operating Procedures for Research Integrity (SOPs4RI) consortium has recommended that both research organizations and funders develop a RIPP. A RIPP outlines the key responsibilities of an organization concerning research integrity and details methods and procedures to foster it. For example, in the case of research organizations, a RIPP should facilitate and stimulate a healthy research environment, proper mentoring and supervision, research ethics structures, research integrity training, high-quality dissemination practices, research collaboration, effective data management, and open and fair procedures to deal with breaches of research integrity [11]. Funders have a different role. They can support, safeguard, and incentivise, or even mandate, responsible research practices from research organizations and researchers. Equally important, funders should make sure that their internal processes live up to the highest standards of research integrity. We recognize that funders are many and varied in their scale, portfolio, disciplinary focus, and the extent to which they have procedures and governance arrangements to support research integrity. For all funders, adopting a RIPP will structure and coordinate research integrity practices, giving clarity and transparency to applicant institutions and researchers.In this Consensus View, we highlight examples of best practice of funders worldwide to foster a research integrity culture. With these examples in mind, we suggest guidelines to support funders in taking a leading role in fostering research integrity. In so doing, we acknowledge the local contexts in which funders operate, but we believe that all funders, large and small, in all parts of the world, can and should contribute to improving research validity and building and maintaining trust in science through incentivising and mandating a culture of research integrity. Our core argument is that developing a tailored RIPP will contribute to building an institutional culture of research integrity, both within funding organizations and among the research organizations and individual researchers they fund. Based on empirical work from the SOPs4RI project, we have identified 6 key research integrity topics: researchers’ compliance with research integrity standards; expectations for research organizations; selection of grant applications; declaration of interests; monitoring funded research; and dealing with internal integrity breaches (Fig 1). We recommend that these topics should be included in a RIPP and provide guidelines on developing and implementing a RIPP.Open in a separate windowFig 1Topics to be covered in a RIPP for funders.An overview of the 6 most important topics identified by the SOPs4RI to be included in research funding organization’s RIPP. RIPP, Research Integrity Promotion Plan; SOPs4RI, Standard Operating Procedures for Research Integrity.     

Fair Resource Allocation to Health Research: Priority Topics for Bioethics Scholarship

This article draws attention to the limited amount of scholarship on what constitutes fairness and equity in resource allocation to health research by individual funders. It identifies three key decisions of ethical significance about resource allocation that research funders make regularly and calls for prioritizing scholarship on those topics – namely, how health resources should be fairly apportioned amongst public health and health care delivery versus health research, how health research resources should be fairly allocated between health problems experienced domestically versus other health problems typically experienced by disadvantaged populations outside the funder's country, and how domestic and non‐domestic health research funding should be further apportioned to different areas, e.g. types of research and recipients. These three topics should be priorities for bioethics research because their outcomes have a substantial bearing on the achievement of health justice. The proposed agenda aims to move discussion on the ethics of health research funding beyond its current focus on the mismatch between worldwide basic and clinical research investment and the global burden of disease. Individual funders’ decision‐making on whether and to what extent to allocate resources to non‐domestic health research, health systems research, research on the social determinants of health, capacity development, and recipients in certain countries should also be the focus of ethical scrutiny.     

Inupiat Health and Proposed Alaskan Oil Development: Results of the First Integrated Health Impact Assessment/Environmental Impact Statement for Proposed Oil Development on Alaska’s North Slope

We report on the first Health Impact Assessment (HIA) for proposed oil and gas development in Alaska’s North Slope region. Public health is not generally analyzed in the Environmental Impact Statement (EIS) process in the U.S. We conducted an HIA for proposed oil development within the National Petroleum Reserve - Alaska in response to growing concerns among North Slope Inupiat communities regarding the potential impacts of regional industrial expansion on their health and culture. We employed a qualitative HIA methodology, involving a combination of stakeholder input, literature review, and qualitative analysis, through which we identified potential health effects. The possible health outcomes identified include increases in diabetes and related metabolic conditions as a result of dietary change; rising rates of substance abuse, domestic violence, and suicide; increased injury rates; more frequent asthma exacerbations; and increased exposure to organic pollutant, including carcinogens and endocrine disruptors. There are also potential benefits, including funding for infrastructure and health care; increased employment and income; and continued funding of existing infrastructure. Based on these findings, we recommend a series of public health mitigation measures. This project represents the first formal effort to include a systematic assessment of public health within the U.S. EIS process. The inclusion of public health concerns within an EIS may offer an important and underutilized avenue through which to argue for environmental management strategies that focus on public health, and may offer communities a stronger voice in the EIS process. An erratum to this article can be found at     

The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health

Doubt about the relevance, appropriateness and transparency of peer review has promoted the use of citation metrics as a viable adjunct or alternative in the assessment of research impact. It is also commonly acknowledged that research metrics will not replace peer review unless they are shown to correspond with the assessment of peers. This paper evaluates the relationship between researchers' influence as evaluated by their peers and various citation metrics representing different aspects of research output in 6 fields of public health in Australia. For four fields, the results showed a modest positive correlation between different research metrics and peer assessments of research influence. However, for two fields, tobacco and injury, negative or no correlations were found. This suggests a peer understanding of research influence within these fields differed from visibility in the mainstream, peer-reviewed scientific literature. This research therefore recommends the use of both peer review and metrics in a combined approach in assessing research influence. Future research evaluation frameworks intent on incorporating metrics should first analyse each field closely to determine what measures of research influence are valued highly by members of that research community. This will aid the development of comprehensive and relevant frameworks with which to fairly and transparently distribute research funds or approve promotion applications.     

The Ethics of Research on Less Expensive,Less Effective Interventions: A Case for Analysis

The Kennedy Krieger lead paint study is a landmark case in human experimentation and a classic case in research ethics. In this paper I use the lead paint study to assist in the analysis of the ethics of research on less expensive, less effective interventions. I critically evaluate an argument by Buchanan and Miller who defend both the Kennedy Krieger lead paint study and public health research on less expensive, less effective interventions. I conclude that Buchanan and Miller’s argument is flawed but that does not mean that research designed to find less effective interventions cannot be justified in some situations. Based on my analysis, I suggest questions to ask when considering such research and I offer some principles to guide us. In the process, light is shed on the various debates and issues raised by the lead paint study; e.g. standards of care, researchers’ responsibilities to research subjects, the distinction between treatment and research and the question of what it is that legitimizes public health research. Merle Spriggs is supported by a grant from the Alfred Felton Bequest which is managed by ANZ Trustees.     

Funding and delivering the routine testing of management interventions to improve conservation effectiveness

Evidence-based approaches are key for underpinning effective conservation practice, but major gaps in the evidence of the effectiveness of interventions limit their use. Conservation practitioners could make major contributions to filling these gaps but often lack the time, funding, or capacity to do so properly. Many funders target the delivery of conservation and can be reluctant to fund primary research. We analysed the literature testing the effectiveness of interventions. Of a sample of 1,265 publications published in 2019 that tested conservation interventions, 96% included academics. Only 21% included conservation practitioners, of which just under half were first or last author. A community of conservation funders and practitioners undertook a series of workshops to explore means of improving the quality and quantity of intervention testing. A survey of the suggested proportion of conservation grants that should be allocated to testing intervention effectiveness showed practitioners tended to prefer larger percentages (median 3–6%) than funders (median 1–3%), but the overlap was considerable. Funders can facilitate the testing of interventions through a range of measures, including welcoming applications that incorporate testing, allocating funds to testing, and providing training and support to deliver testing. The funders represented by the authors of this paper have committed to these actions. Practitioners can contribute by committing to routine testing, benefiting from funding allocated specifically to testing, and establishing processes for testing interventions. The organisations of the practitioner authors have committed to test at least one intervention per year and share findings, regardless of outcome. Currently, practitioners rarely lead the testing of conservation actions. We suggest processes by which both funders and practitioners can make this routine. This will not only improve the effectiveness and cost-efficiency of practice, but also make conservation more attractive to funders.     

An assessment of the hospital exemption landscape across European Member States: regulatory frameworks,use and impact

The hospital exemption (HE) (Article 28(2) of Regulation (EC) No 1394/2007; the “ATMP Regulation”) rule allows the invaluable opportunity to provide patients with access to innovative, potentially life-saving treatments in situations of unmet clinical need. Unlicensed, developmental advanced therapy medicinal products (ATMPs) - cell-, gene- or tissue-based therapies - can be used to treat patients under certain conditions. Such products should be produced on a non-routine basis, custom-made for an individual patient under the responsibility of the requesting physician, for use in a hospital setting within the same Member State in which they are manufactured. The HE rule, and the specific requirements permitting its use, is further regulated at the Member State level, which has led to divergence in the implementation of HE across the European Union (EU). As a result, HE use varies significantly across Member States depending on their respective national legal implementation, policy makers’ interpretation of HE, clarity of guidance at the national level, reimbursement opportunities and level of ATMP research and development activities carried out by academic and commercial organizations. With important variations in how quality, safety and efficacy standards are implemented and controlled across EU Member States for ATMPs provided via the HE rule and a lack of transparency around its use, the HE rule draws concern around its potential impact on public health. In this article, the authors report results of a legal analysis of the implementation of HE across the UK, France, Germany, Italy, Spain, Poland and the Netherlands and research findings on its current utilization, highlighting divergences across countries as well as gaps in legislation and control in these countries. The significance of these divergences and the differing levels of enforcement are discussed as well as their associated impact on patients, industry and health care professionals.     

Enhancing Research Ethics Review Systems in Egypt: The Focus of an International Training Program Informed by an Ecological Developmental Approach to Enhancing Research Ethics Capacity

Recently, training programs in research ethics have been established to enhance individual and institutional capacity in research ethics in the developing world. However, commentators have expressed concern that the efforts of these training programs have placed ‘too great an emphasis on guidelines and research ethics review’, which will have limited effect on ensuring ethical conduct in research. What is needed instead is a culture of ethical conduct supported by national and institutional commitment to ethical practices that are reinforced by upstream enabling conditions (strong civil society, public accountability, and trust in basic transactional processes), which are in turn influenced by developmental conditions (basic freedoms of political freedoms, economic facilities, social opportunities, transparency guarantees, and protective security). Examining this more inclusive understanding of the determinants of ethical conduct enhances at once both an appreciation of the limitations of current efforts of training programs in research ethics and an understanding of what additional training elements are needed to enable trainees to facilitate national and institutional policy changes that enhance research practices. We apply this developmental model to a training program focused in Egypt to describe examples of such additional training activities.     

The Question of Data Integrity in Article-Level Metrics

Interest in and use of article-level metrics (ALMs) has grown rapidly amongst the research community, by researchers, publishers, funders, and research institutions. As this happens, it is critical to ensure secure and reliable data that is trustworthy and can be used by all. Two case studies are presented, which illustrate different approaches to establishing ALM data integrity.     

The primary reasons behind data sharing,its wider benefits and how to cope with the realities of commercial data

Data availability expectations have changed over the years in scientific publishing, nowhere more so than in the field of genomics. This field has spearheaded openness and transparency via public and structured deposition of data. BMC Genomics strongly encourages deposition and unrestricted availability of all primary data underlying research studies both as a way of ensuring reproducibility and standardisation, but also as part of overall community-driven expectation on data deposition and sharing.With funders and publishers moving towards more explicit mandates (regarding data availability), we examined the current barriers to unrestricted availability of data and explored different scenarios in which commercial agreements might run contrary to scientific convention and data sharing policies. In this editorial, Ross Tellam (CSIRO, Australia), Paul Rushton (Texas A&M AgriLife Research) and Peter Schuerman (University of California, Merced), give their views on the importance of data sharing and examine the current challenges in research fields like crop and livestock genomics, where often it is necessary to integrate the interests of academic and commercial stakeholders. We discuss the current approaches, highlight the importance of community-driven standards, and propose ways forward.     

The cost in calculating transport noise disturbances in public decision making

The public decision-making methods in transport are based on cost-benefit analysis, by which the consequences of the decision (standards for vehicles, new infrastructures...) are converted in monetary amounts and compared to the cost of implementation of the decision. But some of these consequences, especially those related to environment, are not directly expressed in monetary terms. The article aims at offsetting this difficulty in the case of noise. The possible methods for getting money values of noise are presented; it is shown that the estimates to which they lead are coherent and consistent. Then a comparison is made between the present procedures and the procedures which could be implemented, and it is shown that large gains of efficiency could be obtained.     

Recovering endangered species and restoring ecosystems: conservation planning for the twenty-first century in the United States

The Endangered Species Act of 1973, as amended, is one of the most significant pieces of conservation legislation ever passed in the United States. The passage of this act spawned the creation of the Endangered Species Program of the US. Fish and Wildlife Service (USFWS). Under the Act, the USFWS has responsibility for implementing many provisions of the Act: listings, consultations, enforcement of prohibitions and recovery planning. Recovering threatened and endangered species, as well as other declining species yet to be listed, is accomplished through many of the activities that occur under the auspices of the Act, not only through the formal recovery planning process. The Act is fundamentally an instrument for ecosystem conservation, although this aspect of the Act is often over-looked. Planning and implementing an ecosystem approach to conservation activities is a priority for the USFWS, not only for threatened and endangered species but for all wildlife. The recent emphasis on regional habitat conservation planning and the development of regional and multi-species recovery plans are indicative of the priority placed on sound ecosystem conservation planning. All of these processes are implemented with the participation of the potentially affected communities and state wildlife management agencies through a public review process. State conservation agencies are part of the process through a special grant programme.     

An analysis of preliminary and post-discussion priority scores for grant applications peer reviewed by the Center for Scientific Review at the NIH

There has been the impression amongst many observers that discussion of a grant application has little practical impact on the final priority scores. Rather the final score is largely dictated by the range of preliminary scores given by the assigned reviewers. The implication is that the preliminary and final scores are the same and the discussion has little impact. The purpose of this examination of the peer review process at the National Institutes of Health is to describe the relationship between preliminary priority scores of the assigned reviewers and the final priority score given by the scientific review group. This study also describes the practical importance of any differences in priority scores. Priority scores for a sample of standard (R01) research grant applications were used in this assessment. The results indicate that the preliminary meeting evaluation is positively correlated with the final meeting outcome but that they are on average significantly different. The results demonstrate that discussion at the meeting has an important practical impact on over 13% of the applications.