Double-blind review favours increased representation of female authors

Double-blind peer review, in which neither author nor reviewer identity are revealed, is rarely practised in ecology or evolution journals. However, in 2001, double-blind review was introduced by the journal Behavioral Ecology. Following this policy change, there was a significant increase in female first-authored papers, a pattern not observed in a very similar journal that provides reviewers with author information. No negative effects could be identified, suggesting that double-blind review should be considered by other journals.     

Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process.Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.We analyzed reviewer scoring data from a research program that had recently modified the review setting from face-to-face to a teleconference format with minimal changes to the overall review procedures. This analysis included approximately 1600 applications over a 4-year period: two years of face-to-face panel meetings compared to two years of teleconference meetings. The average overall scientific merit scores, score distribution, standard deviations and reviewer inter-rater reliability statistics were measured, as well as reviewer demographics and length of time discussing applications.The data indicate that few differences are evident between face-to-face and teleconference settings with regard to average overall scientific merit score, scoring distribution, standard deviation, reviewer demographics or inter-rater reliability. However, some difference was found in the discussion time.These findings suggest that most review outcome measures are unaffected by review setting, which would support the trend of using teleconference reviews rather than face-to-face meetings. However, further studies are needed to assess any correlations among discussion time, application funding and the productivity of funded research projects.     

The ups and downs of peer review

This article traces the history of peer review of scientific publications, plotting the development of the process from its inception to its present-day application. We discuss the merits of peer review and its weaknesses, both perceived and real, as well as the practicalities of several major proposed changes to the system. It is our hope that readers will gain a better appreciation of the complexities of the process and, when serving as reviewers themselves, will do so in a manner that will enhance the utility of the exercise. We also propose the development of an international on-line training program for accreditation of potential referees.     

The validity of peer review in a general medicine journal

All the opinions in this article are those of the authors and should not be construed to reflect, in any way, those of the Department of Veterans Affairs.

Background

Our study purpose was to assess the predictive validity of reviewer quality ratings and editorial decisions in a general medicine journal.

Methods

Submissions to the Journal of General Internal Medicine (JGIM) between July 2004 and June 2005 were included. We abstracted JGIM peer review quality ratings, verified the publication status of all articles and calculated an impact factor for published articles (Rw) by dividing the 3-year citation rate by the average for this group of papers; an Rw>1 indicates a greater than average impact.

Results

Of 507 submissions, 128 (25%) were published in JGIM, 331 rejected (128 with review) and 48 were either not resubmitted after revision was requested or were withdrawn by the author. Of 331 rejections, 243 were published elsewhere. Articles published in JGIM had a higher citation rate than those published elsewhere (Rw: 1.6 vs. 1.1, p = 0.002). Reviewer quality ratings of article quality had good internal consistency and reviewer recommendations markedly influenced publication decisions. There was no quality rating cutpoint that accurately distinguished high from low impact articles. There was a stepwise increase in Rw for articles rejected without review, rejected after review or accepted by JGIM (Rw 0.60 vs. 0.87 vs. 1.56, p<0.0005). However, there was low agreement between reviewers for quality ratings and publication recommendations. The editorial publication decision accurately discriminated high and low impact articles in 68% of submissions. We found evidence of better accuracy with a greater number of reviewers.

Conclusions

The peer review process largely succeeds in selecting high impact articles and dispatching lower impact ones, but the process is far from perfect. While the inter-rater reliability between individual reviewers is low, the accuracy of sorting is improved with a greater number of reviewers.     

Will technology trample peer review in ecology? Ongoing issues and potential solutions

The classical view of peer review is that it is our primary process for assessing and judging whether research results should be published in a scholarly journal. However, the increased pressure to publish and technological developments are transforming peer review such that it is becoming a system that judges where work is published rather than whether the research is publishable (a ‘where rather than if’ process). Ecology is a field in which publication numbers puts a particular pressure on the review system. In this forum piece, I summarize the issues with the current publication system and discuss how technology is changing it, while suggesting solutions for important prior and ongoing issues with the peer review system. The view explored here is that technological developments (e.g. ease of creating journals, internet sites, storage, data generation, sharing of data and analytical code) will not eliminate peer review per se but will allow for a new set of parameters in which ethics and the optimal use of public funding will play a vital role in the evolution of the review process. Synthesis The number of papers and journals in Ecology has increased dramatically in the past decade. I present a critical overview of our review system and proposes that pressure to publish and technological developments have transformed peer review into a system that decides “where rather than if” papers are publishable. While reviewing the current pressures and factors playing a vital role in the evolution of the review and publication systems, I propose potential solutions to deal with current and future challenges to the peer review and publication systems.     

How Long Is Too Long in Contemporary Peer Review? Perspectives from Authors Publishing in Conservation Biology Journals

Delays in peer reviewed publication may have consequences for both assessment of scientific prowess in academia as well as communication of important information to the knowledge receptor community. We present an analysis on the perspectives of authors publishing in conservation biology journals regarding their opinions on the importance of speed in peer-review as well as how to improve review times. Authors were invited to take part in an online questionnaire, of which the data was subjected to both qualitative (open coding, categorizing) and quantitative analyses (generalized linear models). We received 637 responses to 6,547 e-mail invitations sent. Peer-review speed was generally perceived as slow, with authors experiencing a typical turnaround time of 14 weeks while their perceived optimal review time was six weeks. Male and younger respondents seem to have higher expectations of review speed than females and older respondents. The majority of participants attributed lengthy review times to reviewer and editor fatigue, while editor persistence and journal prestige were believed to speed up the review process. Negative consequences of lengthy review times were perceived to be greater for early career researchers and to have impact on author morale (e.g. motivation or frustration). Competition among colleagues was also of concern to respondents. Incentivizing peer-review was among the top suggested alterations to the system along with training graduate students in peer-review, increased editorial persistence, and changes to the norms of peer-review such as opening the peer-review process to the public. It is clear that authors surveyed in this study viewed the peer-review system as under stress and we encourage scientists and publishers to push the envelope for new peer-review models.     

Cooperation between referees and authors increases peer review accuracy

Peer review is fundamentally a cooperative process between scientists in a community who agree to review each other''s work in an unbiased fashion. Peer review is the foundation for decisions concerning publication in journals, awarding of grants, and academic promotion. Here we perform a laboratory study of open and closed peer review based on an online game. We show that when reviewer behavior was made public under open review, reviewers were rewarded for refereeing and formed significantly more cooperative interactions (13% increase in cooperation, P = 0.018). We also show that referees and authors who participated in cooperative interactions had an 11% higher reviewing accuracy rate (P = 0.016). Our results suggest that increasing cooperation in the peer review process can lead to a decreased risk of reviewing errors.